Exploring Where Intelligence Lives in Management by Geoff Hulten

Discover the significance of the location of intelligence, exploring factors like latency, costs, operations, and execution in various scenarios such as client service, back-end hybrid, and more. Geoff Hulten's insights shed light on the impact of intelligence placement on a range of aspects.

• Intelligence Management
• Geoff Hulten
• Data Latency
• Cost Efficiency
• Client Service

cmurr Follow

Uploaded on Sep 22, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Where Intelligence Lives & Intelligence Management Geoff Hulten

2. Overview Where intelligence lives Intelligence management

3. Places Intelligence Could Live Client Service Back-end Hybrid

4. What does it matter where intelligence lives? Latency in Updating Quality is evolving quickly Problem is evolving quickly Risk of costly mistakes Cost of operation Cost of distributing intelligence Cost of executing intelligence Offline operation Work without Internet? Keep it out of Abuser s hands Latency in Execution Slowing the experience Slowing the action The right answer changes too fast

5. Where Intelligence Lives Lives in Service Lives on Client 1 MB Model Daily Update 100k Users 10 Calls/Day 1 mb x 1 Intelligence Creation 1 mb x 100,000 Intelligence Creation Server Server Telemetry 100kb x 10 X 100,000 Clients Clients Total: 100,000 mb + Telemetry Total: 100,001 mb + compute

6. Places Intelligence can Live Where it Lives Latency in Updating Latency in Execution Cost of Operation Offline? Static in Product Poor Excellent Cheap Yes Client Side Variable Excellent Based on update rate Yes Server-Centric Good Internet Roundtrip Can be high No Back-end Variable Variable Variable Partial Hybrid ?? ?? ?? ??

7. Examples of Where Intelligence Lives Kinect Self-Driving Car Latency in Updating Not Important Latency in Updating Important Latency in Execution Critical Latency in Execution Critical Cost of Operation Not Key Factor Cost of Operation Important (?) Offline Operation Important Offline Operation Critical Solution (?) Client Centric Solution (?) Client Centric Anti-Phishing Sprinkler Controller Latency in Updating Slow Okay Latency in Updating Critical Latency in Execution Not Important Latency in Execution Medium Important Cost of Operation Important (?) Cost of Operation Important Offline Operation Critical Offline Operation Not Important Solution (?) Backend (Cache) Solution (?) Hybrid (ALL) Online Shopping Composition Assistant Latency in Updating Slow Okay Latency in Updating Medium Important Latency in Execution Important Latency in Execution Very Important Cost of Operation Important (?) Cost of Operation Very Important Offline Operation Important Offline Operation Not Important Solution (?) Server / Client Solution (?) Server / Backend

8. Intelligence Management Simple Intelligence Management Process of getting new models deployed safely, repeatably, at scale. >> Copy $NewModelPath $DeployedModelPath >> RestartService.exe Complexity Many models Living multiple places Comes from many sources Interdependencies Risk of: High cost Error prone Hard to understand Paralysis Frequency Hourly for three years ~26,000 times Long-lived Human Systems Staff Turnover Skill set of model deployers

9. Effective Intelligence Management Sanity check the intelligence Simplify Deployment Workflow Support controlled light-up Dealing with Mistakes

10. Sanity Checking Intelligence Intelligence creators should test everything But they might not Automate and manage verification workflow Runtime constraints RAM footprint Prediction perf (across contexts) Training vs Runtime prediction parity Obvious mistakes Verify offline accuracy on independent validation set Mistake distribution similar (not more costly) Business critical contexts Critical sub-populations Compatibility with Runtime Model correctly encoded Metadata (thresholds) present Feature extraction code in sync Models all in sync

11. Deploying and Lighting Up (Online Evaluation) Single Deployment All users see all updates at once Simple Relies on great offline tests Controlled Rollout Several live at once, transition slowly Lets you observe user interactions Overhead to build and manage Risk of costly/hard-to-find mistakes. Adds latency. Silent Intelligence Run two versions at once Ensure online is same as offline Gives time to see new contexts Flighting Intelligence (A/B test) Deploy options, track till one better Connects accuracy to true objective Overhead to build and manage Latency. No interactions. Latency. Hard to confirm small gains.

12. Overriding Problems Override mistakes Heuristics or simple rules Rollback problems Deploy new version quickly Creating and deploying them Store multiple versions in runtime Managing them over time Big Red Switch

13. Summary of Intelligence Management Where intelligence lives Static Client-side Server-centric Backend Hybrid Managing Intelligence Sanity checking Deploying Lighting it up Dealing with mistakes