DECISION INTELLIGENCE | CONVERSATIONAL AGENTS | MIMICKING HUMAN SPEECH AND BEHAVIOR
_edited.png)
DECISION INTELLIGENCE: COMPUTATIONAL LINGUISTIC MODELING AND EVALUATION
[Funded by the National Research Foundation of Korea]
ROLE
Research Assistant
DESCRIPTION
As a Research Assistant of the Human and Artificial Intelligence Research Lab at Yonsei University, I am currently exploring how to assist human collaboration with AI agents for efficient decision-making in corporate settings. With my background in Human-Computer Interaction, I am currently focusing on how to mimic human dialogues for decision-making.
TEAMMATES
PI: Keeheon Lee
Kunhee Ryu, Minje Kim, Arina Svetasheva, Yeajoo Yoo, Hogyun Yoo, Sungjoon Whang
TIMELINE
Sept 2025 - Present
RESEARCH METHODS
Currently replication research
and meta-research
TOTAL RESEARCH FUNDING
Total Funding: 247,180,000 KRW
(approx. $172,986 USD)
Funded by National Research Foundation of Korea
PROJECT INTRODUCTION
This research series aims to explore how decision-making processes could be made more efficient in corporate settings, using a multi-agent system.
In order to advance human-AI collaboration, my research team and I are currently exploring collaborative decision-oriented dialogue, as well as decision-making using Decentralized Partially Observable Markov Decision Process (Dec-POMDP).
INSPIRATIONS
Decision-Oriented Dialogue
for Human-AI Collaboration
Lin, J., Tomlin, N., Andreas, J., & Eisner, J. (2024). Decision-oriented dialogue for human–AI collaboration. Transactions of the Association for Computational Linguistics, 12, 892–911. https://doi.org/10.1162/tacl_a_00679
Introduces decision-oriented dialogues (DOD):
AI assistants collaborate with humans via natural language
to solve complex decision-making problems.
Fig 1. An overview of the three featured collaborative dialogue tasks. Tasks include assignment, planning, and mediation (Jessy Lin et al, 2024).
System Features
Involve multiple agents communicating to reach a joint decision.
Three daily-life domains studied:
-
Assignment: Conference paper-reviewer matching
-
Planning: City itinerary creation
-
Mediation: Coordinating group travel schedules
Assignment Task Observations & Roles
-
Each agent has symmetric roles but different and partial edge information.
-
Scores from a random uniform distribution, scaled positively but unknown between agents to avoid numeric communication.
-
Agents share information via dialogue to infer better matches.
-
Decision proposal, acceptance, and dialogue ends when agreement is reached.
Fig 2. The assignment task. (Jessy Lin et al, 2024).
Planning Task Definition
-
The assistant helps the user build an itinerary across city sites.
-
World state is a fully connected graph where edge weights indicate travel times.
-
User preferences include budget limits and desired amenities, while the assistant knows site features but not the user's private preferences.
-
Goal is to select a travel path balancing these information.
Fig 3. The planning task. (Jessy Lin et al, 2024).
Mediation Task Definition
-
Only the assistant can propose itineraries, and the user must accept or reject based on their preferences.
-
The iterative nature of dialogue allows refining plans through back and forth conversation.
-
The environment supports programmatic queries over sites to aid assistant decisions.
-
Scoring system normalizes itinerary quality for comparison.
Fig 4. The mediation task. (Jessy Lin et al, 2024).
Key Findings
-
Substantial gaps between AI assistants based on GPT-3 and human assistants.
-
Humans consistently achieved higher reward scores with shorter, more efficient dialogues. GPT-3 assistants tended to engage in longer conversations but often failed to make progress toward optimal decisions.
-
Self-play, where AI agents simulate both roles, resulted in even lower performance, indicating the challenge for models to collaborate without a human in the loop.
Language models can solve computer tasks
Kim, G., Baldi, P., & McAleer, S. (2023). Language models can solve computer tasks (arXiv:2303.17491). arXiv. https://doi.org/10.48550/arXiv.2303.17491
Explores how large language models (LLMs) can act as agents to solve computer tasks using natural language instructions.
Fig 5. The agent uses RCI prompting to solve a terminal task (deleting a file ending with ".rb") by first generating a step-by-step plan from the natural language instruction (Kim et al, 2023).
Recursive Criticism and Improvement (RCI):
AI assistants collaborate with humans via natural language to solve
complex decision-making problems.
-
technique for making large language models smarter and more reliable when solving computer tasks.
-
RCI helps models learn from their own mistakes, much like how people revise their work.
-
way for AI models to improve their own answers. Instead of just giving one response, the model reviews its work, finds mistakes, and tries again, just like a human editor. This process is recursive, meaning it can repeat several times to refine the answer. RCI helps models become more accurate and thoughtful, especially on complex tasks.
Key Findings
-
RCI enables LLM agents to solve complex computer tasks with minimal data.
-
Limitations include handling long states and limited action spaces; future work may address these.
-
Can be applied to language-based agents in practical settings.