Essential Work Samples for Evaluating AI Interaction Design Skills

AI interaction design has emerged as a critical discipline at the intersection of user experience and artificial intelligence. As organizations increasingly integrate AI capabilities into their products and services, the need for designers who understand how to create intuitive, effective, and ethical AI interactions has never been greater. These designers must bridge the gap between complex AI systems and human users, creating experiences that feel natural while leveraging the unique capabilities of AI.

Evaluating candidates for AI interaction design roles presents unique challenges. Traditional design portfolios may not adequately demonstrate a candidate's understanding of AI-specific considerations such as prompt engineering, error recovery, expectation management, and designing for different user skill levels. Additionally, AI interaction design requires balancing technical feasibility with user needs—a skill that's difficult to assess through interviews alone.

Work samples provide a window into how candidates approach AI interaction challenges in realistic scenarios. By observing candidates as they plan, design, and refine AI interactions, hiring teams can evaluate not just their final output but their thought process, problem-solving approach, and ability to incorporate feedback. This reveals how candidates balance competing priorities and make design decisions that serve both users and business goals.

The following work samples are designed to evaluate key competencies for AI interaction designers: strategic thinking, critical analysis, technical implementation skills, and practical application of AI interaction principles. Each exercise simulates real-world challenges that AI interaction designers face, providing a comprehensive view of a candidate's capabilities in this specialized field.

Activity #1: AI Interaction Flow Planning

This activity evaluates a candidate's ability to strategically plan a complex AI interaction system. Effective AI interaction designers must understand how to map user journeys that leverage AI capabilities while maintaining user control and trust. This exercise reveals how candidates think about the overall architecture of AI interactions, including conversation flows, decision points, and fallback mechanisms.

Directions for the Company:

• Provide the candidate with a brief describing a new AI assistant feature for a specific product (e.g., an AI shopping assistant for an e-commerce platform, an AI research assistant for a knowledge management tool, or an AI coaching feature for a fitness app).
• Include key business objectives, target user personas, and technical constraints in the brief.
• Allow candidates 24-48 hours to prepare their response.
• During the interview, give candidates 15-20 minutes to present their interaction flow, followed by 10 minutes of questions.
• Prepare questions that probe the candidate's reasoning behind key design decisions.

Directions for the Candidate:

• Design a comprehensive interaction flow for the AI feature described in the brief.
• Create a visual representation of the conversation/interaction flow (flowchart, sequence diagram, or state machine).
• Identify key decision points where the AI needs to determine user intent or next steps.
• Include error handling and recovery paths for common failure modes.
• Prepare to explain your design decisions and how they balance user needs with technical capabilities.
• Be ready to discuss how your design handles ambiguity and maintains user trust.

Feedback Mechanism:

• After the presentation, provide one piece of positive feedback about an aspect of the flow that effectively addresses user needs.
• Offer one suggestion for improvement, such as an edge case the candidate didn't consider or a potential friction point in the interaction.
• Give the candidate 5 minutes to verbally explain how they would modify their design based on this feedback.
• Evaluate both their initial design and their ability to quickly incorporate feedback and iterate.

Activity #2: AI Interaction Critique and Redesign

This exercise tests a candidate's ability to critically analyze existing AI interactions and propose improvements. Strong AI interaction designers must be able to identify patterns that create friction or confusion in AI systems and apply design principles to resolve these issues. This activity reveals a candidate's analytical skills and practical knowledge of AI interaction best practices.

Directions for the Company:

• Select an existing AI interaction that has room for improvement (e.g., a chatbot conversation, voice assistant interaction, or AI-powered feature in an application).
• Capture screenshots or recordings of the interaction, including any problematic aspects.
• Prepare a brief explaining the context and goals of the interaction.
• Allow candidates 30 minutes during the interview to review the materials and prepare their critique.
• Have a device available for the candidate to sketch or prototype their redesign ideas.

Directions for the Candidate:

• Review the provided AI interaction example and identify strengths and weaknesses in the current design.
• Analyze the interaction from multiple perspectives: usability, clarity, efficiency, error handling, and appropriate use of AI capabilities.
• Identify 3-5 specific improvements that would enhance the interaction.
• Create rough sketches or wireframes showing your proposed redesign.
• Be prepared to explain the reasoning behind your critique and how your redesign addresses the identified issues.
• Consider both immediate improvements and longer-term strategic changes.

Feedback Mechanism:

• After the candidate presents their critique and redesign, acknowledge one insightful observation they made about the existing interaction.
• Challenge one aspect of their redesign by presenting a new constraint or consideration they may not have factored in.
• Give the candidate 5-10 minutes to revise their approach based on this new information.
• Evaluate their flexibility and problem-solving approach when faced with new constraints.

Activity #3: Error Recovery and Feedback Design

This activity assesses a candidate's ability to design effective error handling and feedback mechanisms for AI interactions. How an AI system responds when it fails or lacks confidence is critical to maintaining user trust and engagement. This exercise reveals how candidates approach these challenging but essential aspects of AI interaction design.

Directions for the Company:

• Prepare a scenario involving an AI system that needs to handle various types of errors or uncertainty (e.g., a language model that can't answer certain questions, a recommendation system with incomplete data, or a voice assistant that misunderstands commands).
• Create a list of specific error scenarios the candidate should address.
• Provide information about the target users and their likely expectations.
• Allow 45-60 minutes for this exercise during the interview.
• Have design tools available (digital or analog) for the candidate to create their designs.

Directions for the Candidate:

• Design error messages and recovery flows for each scenario provided.
• Create a framework for how the AI system should communicate different levels of confidence in its responses.
• Design feedback mechanisms that help users understand the AI's capabilities and limitations.
• Consider how to maintain user trust while being transparent about the system's limitations.
• Prepare mockups or wireframes showing your proposed error handling and feedback designs.
• Be ready to explain how your designs balance honesty about AI limitations with maintaining a positive user experience.

Feedback Mechanism:

• After reviewing the candidate's designs, highlight one particularly effective error handling approach they created.
• Suggest one area where their design might create confusion or frustration for users.
• Ask the candidate to revise one specific error message or recovery flow based on your feedback.
• Evaluate how well they incorporate the feedback while maintaining the overall integrity of their design system.

Activity #4: Prompt Design and Refinement

This exercise evaluates a candidate's ability to design effective prompts for AI systems and refine them based on results. Prompt engineering is becoming a crucial skill for AI interaction designers, as it shapes how AI systems interpret and respond to user inputs. This activity reveals a candidate's understanding of how prompt design influences AI behavior and user experience.

Directions for the Company:

• Set up access to an AI system that the candidate can interact with (e.g., a GPT model, a text-to-image generator, or another AI tool relevant to your products).
• Prepare a specific task that requires careful prompt design (e.g., creating a customer service response, generating content in a specific style, or solving a particular problem).
• Define clear success criteria for what a good output looks like.
• Allow 30-45 minutes for this exercise during the interview.
• Ensure the candidate can see the results of their prompts in real-time.

Directions for the Candidate:

• Design a series of prompts for the AI system to accomplish the specified task.
• Test your prompts with the provided AI system and observe the results.
• Iteratively refine your prompts based on the outputs you receive.
• Document your prompt iterations and the reasoning behind each change.
• Create a final "prompt template" that could be used consistently for similar tasks.
• Be prepared to explain your approach to prompt design and how you addressed any challenges you encountered.

Feedback Mechanism:

• After the candidate presents their final prompt and results, acknowledge one effective technique they used in their prompt design.
• Suggest one way their prompt could be improved to better handle edge cases or produce more consistent results.
• Give the candidate 5-10 minutes to implement your suggestion and test the revised prompt.
• Evaluate their understanding of how small changes in prompt design can significantly impact AI outputs.

Frequently Asked Questions

How long should we allocate for these work samples in our interview process?

Each of these exercises requires 30-60 minutes to complete properly. We recommend selecting 1-2 exercises that best align with your specific needs rather than attempting all four. Consider spreading them across different interview stages or combining a take-home component with an in-person discussion.

Should candidates have access to AI tools during these exercises?

For the Prompt Design activity, access to an actual AI system is essential. For the other activities, AI tools can be helpful but aren't strictly necessary. The focus is on the candidate's design thinking rather than their ability to use specific tools. However, providing access to basic design tools (digital or analog) is recommended.

How technical do candidates need to be to complete these exercises?

These exercises focus on interaction design rather than technical implementation. Candidates should understand AI capabilities and limitations but don't need programming skills. That said, candidates with some technical knowledge of how AI systems work will likely produce more feasible designs.

How should we evaluate candidates who take different approaches to these exercises?

There's rarely one "correct" solution to these design challenges. Evaluate candidates on their process, reasoning, and how well their solution addresses the core requirements. Look for evidence of systematic thinking, user-centered design principles, and an understanding of AI-specific considerations rather than adherence to a particular approach.

Can these exercises be adapted for junior candidates with less AI experience?

Yes, these exercises can be modified for different experience levels. For junior candidates, provide more structure and guidance, focus on fundamental interaction patterns, and evaluate their potential and learning mindset rather than expecting comprehensive solutions. Consider pairing the exercise with educational resources about AI interaction patterns.

How can we ensure these exercises don't disadvantage candidates from diverse backgrounds?

Be mindful that access to and experience with AI tools varies widely. Provide clear instructions and necessary resources, allow adequate preparation time, and focus evaluation on design thinking rather than familiarity with specific AI systems. Consider offering alternatives or accommodations when needed, and ensure your evaluation criteria don't inadvertently favor certain backgrounds or experiences.

AI interaction design is rapidly evolving as AI capabilities advance and user expectations shift. The work samples outlined above provide a structured approach to evaluating candidates' abilities to design intuitive, effective, and ethical AI interactions. By observing how candidates approach these realistic scenarios, you'll gain valuable insights into their design thinking, problem-solving skills, and understanding of AI-specific considerations.

For more resources to help build your AI design team, explore Yardstick's comprehensive hiring tools, including AI-optimized job descriptions, an AI interview question generator, and AI interview guide generator.

Ready to build a complete interview guide for AI interaction design roles? Sign up for a free Yardstick account today!