Sequential Decision Analytics Framework Overview
A Unified Framework for
Sequential Decision Analytics
Olin Business School
University of Washington, St. Louis
November 5-6, 2019
Warren B. Powell
Princeton University
Department of Operations Research
and Financial Engineering
Guidelines for choosing policies
From the real-world to the model
Choosing a policy
Robust cost function
approximation
Lookahead policy
Policy function
approximation
Policy based on value
function approximation
Slide 3
Choosing a policy
•
Policy function approximations
»
PFAs are best for low-
dimensional problems where the
structure of the policy is apparent
from the problem.
»
Potential architectures:
•
Lookup table rules (this was AI in the 1970’s)
•
Linear models?
•
Nonlinear models?
•
Neural networks?
Slide 4
Choosing a policy
•
Cost function approximations
»
CFAs work for both low-and
high-dimensional problems,
where we again have strong
intuition of how uncertainty
would affect the solution.
»
Parameterizations:
•
Schedule slack?
•
Buffer stock?
•
Parameterized forecasts?
Slide 5
Choosing a policy
•
Value function approximations
»
VFAs work best when the
lookahead model is easy to
approximate
»
The issue is not dimensionality –
it is structure.
Can you approximate this as a function???
Slide 6
Choosing a policy
•
Direct lookahead approximations
»
Direct lookahead models should be
used only when all else fails (which
is often). DLAs are important
when decisions now depend on
future plans and forecasts.
»
Question: How are you going to
compute downstream costs?
•
Deterministic? Scenario trees?
•
Simulating a parametric policy?
•
MCTS?
Slide 7
Choosing a policy
•
Don’t forget about hybrids:
»
Parameterized policies
•
Parameterized myopic policies
•
Parameterized lookaheads
»
Lookahead plus VFA
•
May help to shorten the horizon
»
Hybrid PFA with anything
•
PFAs allow you to specify the decision you want, but the logic
may not be very sophisticated.
•
CFAs, VFAs and lookaheads introduced more sophisticated
behaviors.
»
VFA plus tuning
•
Fit the VFA, then tune the VFA so it works better.
Available at
jungle.princeton.edu
Choosing a policy
•
My notes:
»
Model first, then design policies.
»
Think seriously about at least one policy from the policy
search class, and one from the lookahead class.
»
The research community tends to prefer the lookahead
class.
»
Practitioners prefer the policy search class – they are
simpler and make it easier to exploit domain knowledge.
Don’t ignore domain knowledge!
The more “sophisticated
policies” may underperform these simpler methods.
»
But remember…
The price of simplicity is tunable parameters!
Modeling sequential decision problems
•
First build your model
»
Objective function
»
Policy
»
Constraints at time t
»
Transition function
»
Exogenous information
•
Then design your policies:
»
PFA? Exploit obvious
problem structure.
»
CFA? Can you tune a
deterministic approximation
to make it work better?
»
VFA? Can you approximate
the value of being in a
downstream state?
»
Lookahead? Do you have a
forecast? What is the nature
of the uncertainty?
»
Hybrid?
Slide 11
This content discusses a unified framework for sequential decision analytics presented at Olin Business School, University of Washington. Topics include guidelines for choosing policies, policy function approximations, cost function approximations, value function approximations, and direct lookahead approximations. It highlights the importance of different approximations and models in decision-making processes. The discussion emphasizes the need for a structured approach in policy selection and provides insights into various decision-making strategies.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
A Unified Framework for Sequential Decision Analytics Olin Business School University of Washington, St. Louis November 5-6, 2019 Warren B. Powell Princeton University Department of Operations Research and Financial Engineering
Guidelines for choosing policies From the real-world to the model
Choosing a policy Robust cost function approximation Lookahead policy Policy function approximation Policy based on value function approximation Slide 3
Choosing a policy Policy function approximations PFAs are best for low- dimensional problems where the structure of the policy is apparent from the problem. Potential architectures: Lookup table rules (this was AI in the 1970 s) Linear models? Nonlinear models? Neural networks? Slide 4
Choosing a policy Cost function approximations CFAs work for both low-and high-dimensional problems, where we again have strong intuition of how uncertainty would affect the solution. ( ) = ( ) argmin X c x t x tdl l tdl d l Parameterizations: Schedule slack? Buffer stock? Parameterized forecasts? Slide 5
Choosing a policy Value function approximations VFAs work best when the lookahead model is easy to approximate The issue is not dimensionality it is structure. Can you approximate this as a function??? T t = + * t ( ) S argmax ( , ) t C S x E max E ( , ( ))| | , X C S X S S S x + ' ' ' 1 t x t t t t t t t = + t ' 1 t ???(???) Slide 6
Choosing a policy Direct lookahead approximations Direct lookahead models should be used only when all else fails (which is often). DLAs are important when decisions now depend on future plans and forecasts. Question: How are you going to compute downstream costs? Deterministic? Scenario trees? Simulating a parametric policy? MCTS? ?+? ?( ??? , ??? ?( ??? |?))| ??,?+1 |??,?? ?(??,??) + ? ? ?????(??|?) = argmax?? ? =?+1 Slide 7
Choosing a policy Don t forget about hybrids: Parameterized policies Parameterized myopic policies Parameterized lookaheads Lookahead plus VFA May help to shorten the horizon Hybrid PFA with anything PFAs allow you to specify the decision you want, but the logic may not be very sophisticated. CFAs, VFAs and lookaheads introduced more sophisticated behaviors. VFA plus tuning Fit the VFA, then tune the VFA so it works better.
Choosing a policy My notes: Model first, then design policies. Think seriously about at least one policy from the policy search class, and one from the lookahead class. The research community tends to prefer the lookahead class. Practitioners prefer the policy search class they are simpler and make it easier to exploit domain knowledge. Don t ignore domain knowledge!The more sophisticated policies may underperform these simpler methods. But remember The price of simplicity is tunable parameters!
Modeling sequential decision problems Then design your policies: PFA? Exploit obvious problem structure. First build your model Objective function T E = ( ) t min , ( ) S C S X t t t 0 t CFA? Can you tune a deterministic approximation to make it work better? Policy X : X S Constraints at time t VFA? Can you approximate the value of being in a downstream state? = X ( ) S x X t t t t Lookahead? Do you have a forecast? What is the nature of the uncertainty? Transition function S S + = ( ) M , , S x W + 1 1 t t t t Exogenous information ( , ,..., W W Hybrid? ) W 1 2 T Slide 11
