A Wizard of Oz Study Simulating API Usage Dialogues with a Virtual Assistant

Table of Contents

• 1. Introduction
• 2. Methodology & Experimental Design
  • 2.1. Wizard of Oz Protocol
  • 2.2. Participant Recruitment & Tasks
  • 2.3. Data Collection & Annotation Framework
• 3. Results & Key Findings
  • 3.1. Dialogue Act Analysis
  • 3.2. Statistical Overview
  • 3.3. Core Insights from Interactions
• 4. Technical Framework & Mathematical Model
• 5. Analysis Framework: Example Case Study
• 6. Application Outlook & Future Directions
• 7. References
• 8. Original Analysis & Expert Commentary

1. Introduction

Virtual assistants (VAs) are transforming human-computer interaction, yet their application in specialized domains like software engineering remains limited. A primary bottleneck is the scarcity of high-quality, domain-specific dialogue datasets required to train the underlying AI models. This paper addresses this gap by presenting a Wizard of Oz (WoZ) study designed to simulate and collect dialogues between programmers and a virtual assistant for API usage. The study involved 30 professional programmers who believed they were interacting with an AI, while in reality, human experts (“wizards”) generated the responses. The resulting corpus was annotated across multiple dimensions to understand the structure and intent of help-seeking dialogues in a programming context.

2. Methodology & Experimental Design

The core of this research is a meticulously designed WoZ experiment, a proven method in HCI for simulating intelligent systems before they are fully built.

2.1. Wizard of Oz Protocol

The WoZ paradigm was employed to create a believable simulation of a functional API assistant. Programmers interacted via a chat interface, unaware that responses were crafted in real-time by human experts behind the scenes. This method allows for the collection of naturalistic dialogue data that reflects genuine user needs and strategies, which is crucial for training future AI systems, as emphasized in foundational dialogue system literature like that by Rieser and Lemon.

2.2. Participant Recruitment & Tasks

The study recruited 30 professional programmers. Each participant was assigned programming tasks requiring the use of two distinct APIs. The tasks were designed to be non-trivial, prompting the need for assistance and thus generating a rich dialogue corpus.

2.3. Data Collection & Annotation Framework

The collected dialogues were annotated along four key dimensions:

1. Illocutionary Intent: The speaker's goal (e.g., request, inform, confirm).
2. API Information Type: The category of information sought (e.g., syntax, parameter, example).
3. Backward-facing Function: How an utterance relates to previous dialogue (e.g., answer, elaboration).
4. Traceability to API Components: Mapping dialogue elements to specific API classes/methods.

This multi-faceted annotation schema provides a deep, structured understanding of the dialogue flow.

3. Results & Key Findings

3.1. Dialogue Act Analysis

The annotation revealed a diverse range of dialogue acts. Programmers frequently issued complex, multi-part requests that combined questions about syntax, semantics, and usage examples. The “wizard” responses often needed to decompose these requests and provide structured, step-by-step information, highlighting the need for advanced dialogue management in future VAs.

3.2. Statistical Overview

While the paper does not provide exhaustive raw counts, it indicates that the corpus is substantial and varied enough to support machine learning. The distribution of acts across the four annotation dimensions provides a quantitative basis for modeling dialogue state and policy in a virtual assistant.

3.3. Core Insights from Interactions

Key Insight 1: Programmers' help-seeking behavior is highly contextual and iterative, not a simple Q&A.
Key Insight 2: Successful assistance requires linking abstract questions to concrete, traceable API components.
Key Insight 3: The dialogue strategies observed are foundational for designing the conversation logic of an AI-powered assistant.

4. Technical Framework & Mathematical Model

The research implicitly aligns with a Partially Observable Markov Decision Process (POMDP) model common in dialogue systems. The assistant's goal is to choose an action $a$ (e.g., provide an example, ask for clarification) based on its belief state $b(s)$ over the true user state $s$ (e.g., user's knowledge gap, current task step) to maximize a reward $R$ (e.g., task completion).

The belief update can be modeled as: $b'(s') = \eta \cdot O(o | s', a) \sum_{s \in S} T(s' | s, a) b(s)$ where $T$ is the transition function, $O$ is the observation function (interpreting user utterance $o$), and $\eta$ is a normalization constant. The annotated corpus provides the data to learn these functions $T$ and $O$ for the API domain.

5. Analysis Framework: Example Case Study

Scenario: A programmer is trying to use an API method DataFrame.merge() but encounters an error.
Dialogue Snippet (Annotated):

• User: "My merge is failing with a key error. How do I specify the join keys?"
  • Intent: Request
  • Info Type: Syntax/Parameter
  • Traceability: DataFrame.merge(), `on`/`left_on`/`right_on` parameters
• Wizard/Assistant: "The `merge()` method can use the `on`, `left_on`, and `right_on` parameters. If your DataFrames have a common column name, use `on='column_name'`. If they are different, use `left_on` and `right_on`. Can you show me the column names of your two DataFrames?"
  • Intent: Inform + Elicit
  • Info Type: Explanation + Example Prompt
  • Backward Function: Answer + Elaboration

This example shows the multi-turn, information-eliciting strategy necessary for effective assistance.

6. Application Outlook & Future Directions

Short-term: The dataset is a direct training resource for building prototype API assistants using sequence-to-sequence or transformer-based models (e.g., fine-tuning models like Codex or CodeT5).
Medium-term: Integration into Integrated Development Environments (IDEs) as a proactive help panel, reducing context-switching to documentation.
Long-term & Future Research:

• Personalization: Modeling a programmer's expertise level to tailor explanations.
• Multi-modal Assistance: Combining dialogue with code generation, like GitHub Copilot, but with explanatory capabilities.
• Cross-API Generalization: Developing models that can learn transferable help strategies across different libraries and frameworks, moving beyond single-API training.
• Explainable AI for Code: Using the dialogue structure to make code generation models' suggestions more interpretable.

7. References

1. McTear, M., Callejas, Z., & Griol, D. (2016). The Conversational Interface: Talking to Smart Devices. Springer.
2. Rieser, V., & Lemon, O. (2011). Reinforcement Learning for Adaptive Dialogue Systems: A Data-driven Methodology for Dialogue Management and Natural Language Generation. Springer.
3. Serban, I. V., et al. (2015). A survey of available corpora for building data-driven dialogue systems. arXiv preprint arXiv:1512.05742.
4. OpenAI. (2021). Codex. [https://openai.com/blog/openai-codex]
5. Google AI. (2021). Conversational AI. [https://ai.google/research/teams/language/conversational-ai]
6. Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374.

8. Original Analysis & Expert Commentary