Microsoft 365 Copilot
Evaluation, experimentation, query quality, and agentic feature development for enterprise AI chat experiences.
Applied Science · Evaluation · AI Agents
Evaluating agentic systems from controlled benchmarks to deployed behavior.
I’m Alan Delgado de Oliveira, a Senior Applied Scientist working on Microsoft 365 Copilot. My work combines the implementation of agentic features with offline evaluation, controlled online experiments, and longitudinal analysis across progressive deployment stages.
Seattle metropolitan areaOriginally from Rio Grande do Sul, Brazil.
01 / Research profile
Applied science for agentic systems.
I study how agentic behavior can be specified, measured, and improved under realistic conditions, with particular attention to experimental validity, uncertainty, and failure analysis.
Ph.D. in Operations Research
Peer-reviewed research spanning stochastic programming, financial optimization, scenario generation, and decision models.
Agents, from zero to hero
A hands-on Portuguese-language course that takes learners from agent foundations to evaluation, guardrails, and multi-agent systems.
AI Agents
Responsible AI
Natural Language Processing
Machine Learning
Stochastic Programming
Operations Research
Python
PyTorch
Offlineevaluation suites, representative datasets, rubrics, and calibrated automated judges
OnlineA/B experiments that connect quality hypotheses to observed user behavior
Progressiveevidence read across deployment rings before broad product exposure
Ph.D.Operations Research and optimization under uncertainty
02 / Practice
Evaluation methodology.
My work treats evaluation as a sequence of complementary measurement problems, from controlled task-level analysis to randomized experiments and progressive deployment.
Agentic features, end to end
Task scoping, tool and skill design, grounding, retrieval, failure handling, guardrails, and instrumentation.
Offline evaluation design
Representative datasets, clear rubrics, human labeling, judge calibration, slice analysis, and tests for common failure modes.
Online experimentation
A/B tests designed around answerable questions, guardrail metrics, novelty effects, statistical uncertainty, and user-visible outcomes.
Progressive rollout evaluation
Sequencing evidence across deployment rings so small-population signals are interpreted carefully and issues surface early.
Query and retrieval quality
Improving what an AI system asks for and retrieves before optimizing the answer it generates.
Research to product decisions
Translating methods from papers and prototypes into repeatable decision procedures that product teams can run on schedule.
03 / Experience
From optimization research to production AI.
A career built around translating advanced algorithms and quantitative methods into products, evaluation systems, and decisions that work at scale.
2024 — PRESENT
Senior Applied Scientist
Microsoft · Microsoft 365 Copilot
I design evaluation and experimentation approaches for agentic chat features: offline suites that estimate quality, online experiments that test user-visible outcomes, and staged analysis across deployment rings that supports decisions about further exposure, revision, or rollback. I also work on query and retrieval quality upstream of generation.
2021 — 2024
Data & Applied Scientist
Microsoft · Search and User Understanding
I developed deep-learning and NLP capabilities for search relevance and virtual assistants, including semantic models and international-market product expansion.
2019 — 2020
Data Scientist → Senior Data Scientist
Poatek
I delivered machine-learning systems for credit risk, AutoML, satellite-image segmentation, conversational-assistant analysis, and high-volume optimal sampling.
Since 2019 · PART-TIME
Guest Lecturer, MBA in Business Analytics
Federal University of Rio Grande do Sul
I taught graduate-level courses connecting machine learning, analytics, mathematical modeling, and optimization with practical business problems.
2017 — 2018
Research Assistant & Postdoctoral Fellow
The George Washington University
I developed reformulations and algorithms for stochastic optimization, including air-ambulance allocation models for reliable military medical evacuation.
2015 — 2018
Ph.D. Researcher
Federal University of Rio Grande do Sul
I researched multistage stochastic programming, asset-liability management, scenario generation, and long-term wealth allocation under uncertainty.
Ph.D. · Operations ResearchUFRGS · 2015–2018
Master’s · Quantitative MethodsUFRGS · 2013–2014
B.Sc. · Computer ScienceUFRGS · 2007–2011
04 / Writing
Technical notes and reproducible artifacts.
Concrete treatments of measurement problems, statistical assumptions, implementation choices, and research results relevant to agentic systems.
An executable protocol for agent evaluation
A synthetic end-to-end example combining deterministic task contracts, difference-in-proportions confidence intervals, guardrail non-inferiority, Wilson lower bounds, and ring-level expand/continue/hold rules.
Separating capability, causal effect, and rollout risk
A formal decomposition of three evaluation questions: whether an agent can complete a task under controlled conditions, whether a treatment changes user outcomes, and whether quality remains stable as exposure expands.
Medical evacuation with exogenous and decision-dependent uncertainty
A chance-constrained stochastic optimization model, Boolean reformulation, convexification framework, and public computational experiments for reliable medical-evacuation network design.
05 / Interactive and current
Methods you can inspect.
Two surfaces that go beyond describing methodology: one lets you manipulate the statistics directly, the other tracks the literature the methodology draws on.
Evidence explorer
Adjust offline pass rates, experiment sample sizes, guardrail margins, and ring exposure, and watch confidence intervals, minimum detectable effect, Wilson bounds, and the resulting expand/hold decision update. Includes the cases where a favorable headline number is not decision-grade.
Agent methodology tracks
Foundational papers on evaluation, agent skills and tool use, prompt and context engineering, and experimentation — paired with a filtered view of recent work. Selection logic is public, relevance outranks popularity, and metadata is always fetched rather than transcribed.
06 / Technical implementations
Open methods and software.
Public implementations spanning agent evaluation, stochastic optimization, responsible AI, and natural language processing.
Public methods, synthetic data
Agent Evaluation Playbook
A vendor-neutral reference implementation for evaluating agentic features across offline task suites, online A/B experiments, and progressive deployment rings. Includes transparent statistics, rollout decisions, tests, and an end-to-end report.
View repository ↗Medical evacuation under uncertainty
Public software and experimental data supporting the INFORMS Journal on Computing paper on exogenous and decision-dependent uncertainty.
View code & data ↗Responsible AI framework
A proof of concept for detecting harmful model output using DistilBERT, synthetic data, classification logging, and human review.
View repository ↗Portuguese sentiment analysis
Word2Vec and bag-of-words models trained on 300,000 Portuguese-language tweets.
View repository ↗07 / Teaching
Agentes de IA: Zero to Hero.
A Portuguese-language journey from first principles to building and evaluating practical agents.
Curso em português
6 ×
3h Fundamentos → Sistemas multiagentes
3h Fundamentos → Sistemas multiagentes
Tópicos Especiais em IA
Aprenda construindo um agente de ponta a ponta.
Seis encontros combinam conceitos, código em Python e uma imersão prática. Cada aluno define, implementa, testa, avalia e apresenta um agente de tarefa única.
01Fundamentos, anatomia e ReAct
02Reasoning, planning e tools
03Skills, MCP, RAG e memória
04Falhas, avaliação e observabilidade
05Imersão: escopo, contrato e testes
06Guardrails, logs e demonstração
08 / Research
Selected publications.
Research in optimization, uncertainty, financial decision-making, and operational planning.
2025
Stochastic Optimization Model with Exogenous and Decision-Dependent Uncertainty for Medical Evacuation
Journal article · INFORMS Journal on Computing · with M. A. Lejeune and F. Margot
2025
Prompt and Reliable Medical Evacuation with Air Ambulances
Conference extended abstract · TRISTAN 2025 · with Miguel A. Lejeune and François Margot
2020
Compacting multistage stochastic programming models through a new implicit extensive framework
Journal article · Journal of Computational Science
2020
Solving the index tracking problem based on a convex reformulation for cointegration
Journal article · Finance Research Letters
2018
Performance comparison of scenario-generation methods applied to a stochastic optimization asset-liability management model
Journal article · Pesquisa Operacional
2017
A multistage stochastic programming asset-liability management model
Journal article · Optimization and Engineering
Technical and research interests
Evaluation, experimentation, optimization, and agentic systems.
My current interests include measurement methodology for AI agents, randomized online experiments, retrieval and query quality, stochastic optimization, and technical education.