↓↓↓点击关注，回复资料，10个G的惊喜

来源：DeepHub IMBA
本文约5200字，建议阅读10分钟 本文为你举例介绍MDP的基本概念。

马尔可夫决策过程(Markov decision process, MDP)是人工智能中的一个重要概念，也是强化学习的理论基础之一。在今天的文章中，我们使用来自Stuart Russell和Peter Norvig的《Artificial Intelligence: A Modern Approach》一书中的网格例子来介绍MDP的基本概念。

我们的吃豆人游戏

世界示例

MDP的定义

具有马尔可夫转移模型和附加奖励的完全可观察的随机环境的顺序决策问题称为马尔可夫决策过程或 MDP，由一组状态（具有初始状态 s₀）组成；每个状态下的一组动作；一个转换模型 P(s'| s, a);和奖励函数 R(s)。

网格世界的代码实现

0,0,0,10,3,0,20,0,0,0

import numpy as npimport matplotlib.pyplot as pltimport matplotlib.patches as patches

class GridWorld:  def __init__(self, filename, reward=None):      if reward is None:          reward = {0: -0.04, 1: 1.0, 2: -1.0, 3: np.NaN}      file = open(filename)      self.map = np.array(          [list(map(float, s.strip().split(","))) for s in file.readlines()]      )      self.num_rows = self.map.shape[0]      self.num_cols = self.map.shape[1]      self.num_states = self.num_rows * self.num_cols      self.num_actions = 4      self.reward = reward      self.reward_function = self.get_reward_table()      self.transition_model = self.get_transition_model()
  def get_state_from_pos(self, pos):      return pos[0] * self.num_cols + pos[1]
  def get_pos_from_state(self, state):      return state // self.num_cols, state % self.num_cols

奖励函数

class GridWorld:  def get_reward_function(self):      reward_table = np.zeros(self.num_states)      for r in range(self.num_rows):          for c in range(self.num_cols):              s = self.get_state_from_pos((r, c))              reward_table[s] = self.reward[self.map[r, c]]      return reward_table

转换模型

class GridWorld:  def get_transition_model(self, random_rate=0.2):      transition_model = np.zeros((self.num_states, self.num_actions, self.num_states))      for r in range(self.num_rows):          for c in range(self.num_cols):              s = self.get_state_from_pos((r, c))              neighbor_s = np.zeros(self.num_actions)              for a in range(self.num_actions):                  new_r, new_c = r, c                  if a == 0:                      new_r = max(r - 1, 0)                  elif a == 1:                      new_c = min(c + 1, self.num_cols - 1)                  elif a == 2:                      new_r = min(r + 1, self.num_rows - 1)                  elif a == 3:                      new_c = max(c - 1, 0)                  if self.map[new_r, new_c] == 3:                      new_r, new_c = r, c                  s_prime = self.get_state_from_pos((new_r, new_c))                  neighbor_s[a] = s_prime              for a in range(self.num_actions):                  transition_model[s, a, int(neighbor_s[a])] += 1 - random_rate                  transition_model[s, a, int(neighbor_s[(a + 1) % self.num_actions])] += random_rate / 2.0                  transition_model[s, a, int(neighbor_s[(a - 1) % self.num_actions])] += random_rate / 2.0      return transition_model

策略

class GridWorld:def generate_random_policy(self):  return np.random.randint(self.num_actions, size=self.num_states)

策略的执行

class GridWorld:def execute_policy(self, policy, start_pos=(2, 0)):  s = self.get_state_from_pos(start_pos)  total_reward = 0  state_history = [s]  while True:    temp = self.transition_model[s, policy[s]]    s_prime = np.random.choice(self.num_states, p=temp)    state_history.append(s_prime)    r = self.reward_function[s_prime]    total_reward += r    s = s_prime    if r == 1 or r == -1:      break  return total_reward

更多策略

如何找到最优策略？

https://github.com/clumsyhandyman/mad-from-scratch

推荐阅读
决策树可视化，被惊艳到了！
开发机器学习APP，太简单了
200 道经典机器学习面试题总结
卷积神经网络（CNN）数学原理解析
收手吧，华强！我用机器学习帮你挑西瓜
为了这个GIF，我专门建了一个网站
【保姆级教程】白嫖老外的云服务器

三连在看，月入百万👇