Assign Reinforcement Work==

**Reinforcement learning** is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take *actions* in an *environment* so as to maximize some notion of cumulative *reward*. The problem, due to its generality, is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, statistics, and genetic algorithms. In the operations research and control literature, the field where reinforcement learning methods are studied is called *approximate dynamic programming*. The problem has been studied in the theory of optimal control, though most studies there are concerned with existence of optimal solutions and their characterization, and not with the learning or approximation aspects. In economics and game theory, reinforc==Assign Reinforcement Work==

**Reinforcement learning** is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take *actions* in an *environment* so as to maximize some notion of cumulative *reward*. The problem, due to its generality, is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, statistics, and genetic algorithms. In the operations research and control literature, the field where reinforcement learning methods are studied is called *approximate dynamic programming*. The problem has been studied in the theory of optimal control, though most studies there are concerned with existence of optimal solutions and their characterization, and not with the learning or approximation aspects. In economics and game theory, reinforcement learning may be used to explain how equilibrium may arise under bounded rationality.

In machine learning, the environment is typically formulated as a Markov decision process (MDP), and many reinforcement learning algorithms for this context are highly related to dynamic programming techniques. The main difference between the classical techniques and reinforcement learning algorithms is that the latter do not need knowledge about the MDP and they target large MDPs where exact methods become infeasible.

Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge). The exploration vs. exploitation trade-off in reinforcement learning has been most thoroughly studied through the multi-armed bandit problem and in finite MDPs.

Why You Learn More Effectively by Writing Than Typing 249,481 3 Melanie Pinola Filed to: Writing

Mind Hacks Typing Writing tips Republished Top io9

1/21/11 1:00pm

Share to Kinja Share to Facebook Share to Twitter Go to permalink

Why You Learn More Effectively by Writing Than TypingExpand Ads by Google PsPrint - Official Site We Guarantee the Lowest Price! Color, Quality & Quick Turnaround www.psprint.com

The act of writing helps you clarify your thoughts, remember things better, and reach your goals more surely. Here's a look at the science and psychology behind writing, and why the pen may be mightier than the keyboard. Why You Should Risk Dweebhood with Written Goals

Once you get over the idea that people who walk around with a list of personal goals in their…Read more

Photo by sun dazed.

Many productivity experts and writers have long espoused the power of writing things down (in fact, paper is our many of our favorite to-do list manager and we're a little fanatical about our favorite pens). Best To-Do List Manager: Paper

Last week we asked you to share your favorite to-do list manager, we rounded up the most popular…Read more Why Writing Works

Patrick E. McLean's defense of writing longhand is a poetic dissertation on the subject; words can rush out in their raw, feral state when the pen is your tool. Technology, meanwhile, can be too distracting and distancing. A Defense of Writing Longhand

Some of us love using technology to write, but as we highlighted yesterday, the best writing system …Read more

Maybe you're on the other side of the fence, though, and think all this just a lot of pure romanticism: People may feel more comfortable and productive with pen and paper because that's what they've used most of their lives (and what we as a species have used for centuries), but some like typing more and can do it more quickly. Certainly, more of us are becoming fast typists by necessity and the art of handwriting is deteriorating.

A couple of studies, though, substantiate why the physical act of writing really does boost learning and goal achievement. Hoping to provide actual scientific proof on the efficacy of writing down and sharing goals (to make up for an often-quoted mythical Harvard/Yale study of goals), a psych professor at Dominican University of California found that people who wrote down their goals, shared them with others, and maintained accountability for their goals were 33% more likely to achieve them, versus those who just formulated goals. (One can argue that in this instance, typing would be equally effective; see "Why Writing Works Better Than Typing" below for why writing still may be better.) Another study found positive effects of writing on learning foreign words, and a survey of note-taking studies found several examples where taking notes helped students with recall and academic performance.

## Why You Learn More Effectively by Writing Than TypingEdit

The research results may seem common sense or obvious to many of us. If you're interested in the biology behind writing's effect on our achievements, though, here's a little background: Writing stimulates a bunch of cells at the base of the brain called the reticular activating system (RAS). The RAS acts as a filter for everything your brain needs to process, giving more importance to the stuff that you're actively focusing on at the moment—something that the physical act of writing brings to the forefront. In Write It Down, Make It Happen, author Henriette Anne Klauser says that "Writing triggers the RAS, which in turn sends a signal to the cerebral cortex: ‘Wake up! Pay attention! Don't miss this detail!' Once you write down a goal, your brain will be working overtime to see you get it, and will alert you to the signs and signals that […] were there all along." Why Writing Works Better Than Typing

There may also be a scientific basis for the pen's superiority over the keyboard when it comes to writing development and cognitive functions. Dr. Virginia Berniger, who studies reading and writing systems and their relationship to learning processes, found that children's writing ability was consistently better (they wrote more, faster, and more complete sentences) when they used a pen rather than a keyboard; these are, of course, subjects without a penchant for using either tool. We also previously covered the WSJ article that connected handwriting and cognitive abilities; in one of the studies cited, adults learned new symbols and graphic shapes better when they reproduced them with pen-and-paper instead of typing them. Handwriting Practice Could Improve Your Cognitive Abilities

Typing is certainly a lot more efficient, but discarding handwriting entirely might not be such a…Read more

The difference, Berniger notes, may lie in the fact that with writing, you use your hand to form the letters (and connect them), thereby more actively engaging the brain in the process. Typing, on the other hand, involves just selecting letters by pressing identical-looking keys.

Of course, whether the pen or the keyboard is better for you depends on your personal experience and comfort with these tools. As a compromise, perhaps we should all get stylus-friendly tablet PCs or digital pens. Review: Livescribe Pulse Digital Pen/Recorder (Verdict: It's Good for Notetakers)

## Introduction Edit

The basic reinforcement learning model consists of:

- a set of environment states $ S $;

- a set of actions $ A $;

- rules of transitioning between states;

- rules that determine the
*scalar immediate reward*of a transition; and

- rules that describe what the agent observes.

The rules are often stochastic. The observation typically involves the scalar immediate reward associated with the last transition.

In many works, the agent is also assumed to observe the current environmental state, in which case we talk about *full observability*, whereas in the opposing case we talk about *partial observability*. Sometimes the set of actions available to the agent is restricted (e.g., you cannot spend more money than what you possess).

A reinforcement learning agent interacts with its environment in discrete time steps.

At each time $ t $, the agent receives an observation $ o_t $, which typically includes the reward $ r_t $.

It then chooses an action $ a_t $ from the set of actions available, which is subsequently sent to the environment.

The environment moves to a new state $ s_{t+1} $ and the reward $ r_{t+1} $ associated with the *transition* $ (s_t,a_t,s_{t+1}) $ is determined.

The goal of a reinforcement learning agent is to collect as much reward as possible. The agent can choose any action as a function of the history and it can even randomize its action selection.

When the agent's performance is compared to that of an agent which acts optimally from the beginning, the difference in performance gives rise to the notion of *regret*.

Note that in order to act near optimally, the agent must reason about the long term consequences of its actions: In order to maximize my future income I had better go to school now, although the immediate monetary reward associated with this might be negative.

Thus, reinforcement learning is particularly well suited to problems which include a long-term versus short-term reward trade-off. It has been applied successfully to various problems, including robot control, elevator scheduling, telecommunications, backgammon and checkers (Sutton and Barto 1998, Chapter 11).

Two components make reinforcement learning powerful:

The use of samples to optimize performance and the use of function approximation to deal with large environments.

Thanks to these two key components, reinforcement learning can be used in large environments in any of the following situations:

- A model of the environment is known, but an analytic solution is not available;

- Only a simulation model of the environment is given (the subject of simulation-based optimization);

- The only way to collect information about the environment is by interacting with it.

The first two of these problems could be considered planning problems (since some form of the model is available), while the last one could be considered as a genuine learning problem. However, under a reinforcement learning methodology both planning problems would be converted to machine learning problems.

## Exploration Edit

The reinforcement learning problem as described requires clever exploration mechanisms. Randomly selecting actions, without reference to an estimated probability distribution, is known to give rise to very poor performance. The case of (small) finite MDPs is relatively well understood by now.

However, due to the lack of algorithms that would provably scale well with the number of states (or scale to problems with infinite state spaces), in practice people resort to simple exploration methods. One such method is $ \epsilon $-greedy, when the agent chooses the action that it believes has the best long-term effect with probability $ 1-\epsilon $, and it chooses an action uniformly at random, otherwise. Here, $ 0<\epsilon<1 $ is a tuning parameter, which is sometimes changed, either according to a fixed schedule (making the agent explore less as time goes by), or adaptively based on some heuristics (Tokic & Palm, 2011).

## Algorithms for control learning Edit

Even if the issue of exploration is disregarded and even if the state was observable (which we assume from now on), the problem remains to find out which actions are good based on past experience.

### Criterion of optimality Edit

For simplicity, assume for a moment that the problem studied is *episodic*, an episode ending when some *terminal state* is reached. Assume further that no matter what course of actions the agent takes, termination is inevitable. Under some additional mild regularity conditions the expectation of the total reward is then well-defined, for *any* policy and any initial distribution over the states. Here, a policy refers to a mapping that assigns some probability distribution over the actions to all possible histories.

Given a fixed initial distribution $ \mu $, we can thus assign the expected return $ \rho^\pi $ to policy $ \pi $:

- $ \rho^\pi = E[R|\pi], $

where the random variable $ R $ denotes the *return* and is defined by

- $ R=\sum_{t=0}^{N-1} r_{t+1}, $

where $ r_{t+1} $ is the reward received after the $ t $-th transition, the initial state is sampled at random from $ \mu $ and actions are selected by policy $ \pi $. Here, $ N $ denotes the (random) time when a terminal state is reached, i.e., the time when the episode terminates.

In the case of non-episodic problems the return is often *discounted*,

- $ R=\sum_{t=0}^\infty \gamma^t r_{t+1}, $

giving rise to the total expected discounted reward criterion. Here $ 0 \le \gamma \le 1 $ is the so-called *discount-factor*. Since the undiscounted return is a special case of the discounted return, from now on we will assume discounting. Although this looks innocent enough, discounting is in fact problematic if one cares about online performance. This is because discounting makes the initial time steps more important. Since a learning agent is likely to make mistakes during the first few steps after its "life" starts, no uninformed learning algorithm can achieve near-optimal performance under discounting even if the class of environments is restricted to that of finite MDPs. (This does not mean though that, given enough time, a learning agent cannot figure how to act near-optimally, if time was restarted.)

The problem then is to specify an algorithm that can be used to find a policy with maximum expected return.

From the theory of MDPs it is known that, without loss of generality, the search can be restricted to the set of the so-called *stationary* policies. A policy is called stationary if the action-distribution returned by it depends only on the last state visited (which is part of the observation history of the agent, by our simplifying assumption). In fact, the search can be further restricted to *deterministic* stationary policies. A deterministic stationary policy is one which deterministically selects actions based on the current state. Since any such policy can be identified with a mapping from the set of states to the set of actions, these policies can be identified with such mappings with no loss of generality.

### Brute force Edit

The brute force approach entails the following two steps:

- For each possible policy, sample returns while following it

- Choose the policy with the largest expected return

One problem with this is that the number of policies can be extremely large, or even infinite. Another is that variance of the returns might be large, in which case a large number of samples will be required to accurately estimate the return of each policy.

These problems can be ameliorated if we assume some structure and perhaps allow samples generated from one policy to influence the estimates made for another. The two main approaches for achieving this are value function estimation and direct policy search.

### Value function approaches Edit

Value function approaches attempt to find a policy that maximizes the return by maintaining a set of estimates of expected returns for some policy (usually either the "current" or the optimal one).

These methods rely on the theory of MDPs, where optimality is defined in a sense which is stronger than the above one: A policy is called optimal if it achieves the best expected return from *any* initial state (i.e., initial distributions play no role in this definition). Again, one can always find an optimal policy amongst stationary policies.

To define optimality in a formal manner, define the value of a policy $ \pi $ by

- $ V^{\pi} (s) = E[R|s,\pi], $

where $ R $ stands for the random return associated with following $ \pi $ from the initial state $ s $.

Define $ V^*(s) $ as the maximum possible value of $ V^\pi(s) $, where $ \pi $ is allowed to change:

- $ V^*(s) = \sup \limits_\pi V^{\pi}(s). $

A policy which achieves these *optimal values* in *each* state is called *optimal*. Clearly, a policy optimal in this strong sense is also optimal in the sense that it maximizes the expected return $ \rho^\pi $, since $ \rho^\pi = E[ V^\pi(S) ] $, where $ S $ is a state randomly sampled from the distribution $ \mu $.

Although state-values suffice to define optimality, it will prove to be useful to define action-values. Given a state $ s $, an action $ a $ and a policy $ \pi $, the action-value of the pair $ (s,a) $ under $ \pi $ is defined by

- $ Q^\pi(s,a) = E[R|s,a,\pi],\, $

where, now, $ R $ stands for the random return associated with first taking action $ a $ in state $ s $ and following $ \pi $, thereafter.

It is well-known from the theory of MDPs that if someone gives us $ Q $ for an optimal policy, we can always choose optimal actions (and thus act optimally) by simply choosing the action with the highest value at each state.

The *action-value function* of such an optimal policy is called the *optimal action-value function* and is denoted by $ Q^* $.

In summary, the knowledge of the optimal action-value function *alone* suffices to know how to act optimally.

Assuming full knowledge of the MDP, there are two basic approaches to compute the optimal action-value function, value iteration and policy iteration.

Both algorithms compute a sequence of functions $ Q_k $ ($ k=0,1,2,\ldots, $) which converge to $ Q^* $.

Computing these functions involves computing expectations over the whole state-space, which is impractical for all, but the smallest (finite) MDPs, never mind the case when the MDP is unknown.

In reinforcement learning methods the expectations are approximated by averaging over samples and one uses function approximation techniques to cope with the need to represent value functions over large state-action spaces.

#### Monte Carlo methodsEdit

The simplest Monte Carlo methods can be used in an algorithm that mimics policy iteration.

Policy iteration consists of two steps: *policy evaluation* and *policy improvement*.

The Monte Carlo methods are used in the policy evaluation step.

In this step, given a stationary, deterministic policy $ \pi $, the goal is to compute the function values $ Q^\pi(s,a) $ (or a good approximation to them) for all state-action pairs $ (s,a) $.

Assume (for simplicity) that the MDP is finite and in fact a table representing the action-values fits into the memory.

Further, assume that the problem is episodic and after each episode a new one starts from some random initial state.

Then, the estimate of the value of a given state-action pair $ (s,a) $can be computed by simply averaging the sampled returns which originated from $ (s,a) $ over time.

Given enough time, this procedure can thus construct a precise estimate $ Q $ of the action-value function $ Q^\pi $.

This finishes the description of the policy evaluation step.

In the policy improvement step, as it is done in the standard policy iteration algorithm, the next policy is obtained by computing a *greedy* policy with respect to $ Q $: Given a state $ s $, this new policy returns an action that maximizes $ Q(s,\cdot) $. In practice one often avoids computing and storing the new policy, but uses lazy evaluation to defer the computation of the maximizing actions to when they are actually needed.

A few problems with this procedure are as follows:

- The procedure may waste too much time on evaluating a suboptimal policy;

- It uses samples inefficiently in that a long trajectory is used to improve the estimate only of the
*single*state-action pair that started the trajectory;

- When the returns along the trajectories have
*high variance*, convergence will be slow;

- It works in
*episodic problems only*;

- It works in
*small, finite MDPs only*.

#### Temporal difference methodsEdit

The first issue is easily corrected by allowing the procedure to change the policy (at all, or at some states) before the values settle. However good this sounds, this may be dangerous as this might prevent convergence. Still, most current algorithms implement this idea, giving rise to the class of *generalized policy iteration* algorithm. We note in passing that actor critic methods belong to this category.

The second issue can be corrected within the algorithm by allowing trajectories to contribute to any state-action pair in them.

This may also help to some extent with the third problem, although a better solution when returns have high variance is to use Sutton's temporal difference (TD) methods which are based on the recursive Bellman equation. Note that the computation in TD methods can be incremental (when after each transition the memory is changed and the transition is thrown away), or batch (when the transitions are collected and then the estimates are computed once based on a large number of transitions). Batch methods, a prime example of which is the least-squares temporal difference method due to Bradtke and Barto (1996), may use the information in the samples better, whereas incremental methods are the only choice when batch methods become infeasible due to their high computational or memory complexity. In addition, there exist methods that try to unify the advantages of the two approaches. Methods based on temporal differences also overcome the second but last issue.

In order to address the last issue mentioned in the previous section, *function approximation methods* are used.

In *linear function approximation* one starts with a mapping $ \phi $ that assigns a finite-dimensional vector to each state-action pair. Then, the action values of a state-action pair $ (s,a) $ are obtained by linearly combining the components of $ \phi(s,a) $ with some *weights* $ \theta $:

- $ Q(s,a) = \sum \limits_{i=1}^d \theta_i \phi_i(s,a) $.

The algorithms then adjust the weights, instead of adjusting the values associated with the individual state-action pairs.

However, linear function approximation is not the only choice.

More recently, methods based on ideas from nonparametric statistics (which can be seen to construct their own features) have been explored.

So far, the discussion was restricted to how policy iteration can be used as a basis of the designing reinforcement learning algorithms. Equally importantly, value iteration can also be used as a starting point, giving rise to the Q-Learning algorithm (Watkins 1989) and its many variants.

The problem with methods that use action-values is that they may need highly precise estimates of the competing action values, which can be hard to obtain when the returns are noisy. Though this problem is mitigated to some extent by temporal difference methods and if one uses the so-called compatible function approximation method, more work remains to be done to increase generality and efficiency. Another problem specific to temporal difference methods comes from their reliance on the recursive Bellman equation. Most temporal difference methods have a so-called $ \lambda $ parameter $ (0\le \lambda\le 1) $ that allows one to continuously interpolate between Monte-Carlo methods (which do not rely on the Bellman equations) and the basic temporal difference methods (which rely entirely on the Bellman equations), which can thus be effective in palliating this issue.

### Direct policy search Edit

An alternative method to find a good policy is to search directly in (some subset) of the policy space, in which case the problem becomes an instance of stochastic optimization. The two approaches available are gradient-based and gradient-free methods.

Gradient-based methods (giving rise to the so-called *policy gradient methods*) start with a mapping from a finite-dimensional (parameter) space to the space of policies: given the parameter vector $ \theta $, let $ \pi_\theta $ denote the policy associated to $ \theta $.

Define the performance function by

- $ \rho(\theta) = \rho^{\pi_\theta}. $

Under mild conditions this function will be differentiable as a function of the parameter vector $ \theta $.

If the gradient of $ \rho $ was known, one could use gradient ascent.

Since an analytic expression for the gradient is not available, one must rely on a noisy estimate.

Such an estimate can be constructed in many ways, giving rise to algorithms like Williams' REINFORCE method (which is also known as the likelihood ratio method in the simulation-based optimization literature).

Policy gradient methods have received a lot of attention in the last couple of years (e.g., Peters et al. (2003)), but they remain an active field.

The issue with many of these methods is that they may get stuck in local optima (as they are based on local search).

A large class of methods avoids relying on gradient information.

These include simulated annealing, cross-entropy search or methods of evolutionary computation.

Many gradient-free methods can achieve (in theory and in the limit) a global optimum.

In a number of cases they have indeed demonstrated remarkable performance.

The issue with policy search methods is that they may converge slowly if the information based on which they act is noisy.

For example, this happens when in episodic problems the trajectories are long and the variance of the returns is large. As argued beforehand, value-function based methods that rely on temporal differences might help in this case. In recent years, several actor-critic algorithms have been proposed following this idea and were demonstrated to perform well in various problems.

## Theory Edit

The theory for small, finite MDPs is quite mature.

Both the asymptotic and finite-sample behavior of most algorithms is well-understood.

As mentioned beforehand, algorithms with provably good online performance (addressing the exploration issue) are known.

The theory of large MDPs needs more work. Efficient exploration is largely untouched (except for the case of bandit problems).

Although finite-time performance bounds appeared for many algorithms in the recent years, these bounds are expected to be rather loose and thus more work is needed to better understand the relative advantages, as well as the limitations of these algorithms.

For incremental algorithm asymptotic convergence issues have been settled. Recently, new incremental, temporal-difference-based algorithms have appeared which converge under a much wider set of conditions than was previously possible (for example, when used with arbitrary, smooth function approximation).

## Current research Edit

Current research topics include:

adaptive methods which work with fewer (or no) parameters under a large number of conditions,

addressing the exploration problem in large MDPs,

large-scale empirical evaluations,

learning and acting under partial information (e.g., using Predictive State Representation),

modular and hierarchical reinforcement learning,

improving existing value-function and policy search methods,

algorithms that work well with large (or continuous) action spaces,

transfer learning,

lifelong learning,

efficient sample-based planning (e.g., based on Monte-Carlo tree search).

Multiagent or Distributed Reinforcement Learning is also a topic of interest in current research.

There is also a growing interest in real life applications of reinforcement learning.

Successes of reinforcement learning are collected on

here and

here.

Reinforcement learning algorithms such as TD learning are also being investigated as a model for Dopamine-based learning in the brain. In this model, the dopaminergic projections from the substantia nigra to the basal ganglia function as the prediction error. Reinforcement learning has also been used as a part of the model for human skill learning, especially in relation to the interaction between implicit and explicit learning in skill acquisition (the first publication on this application was in 1995-1996, and there have been many follow-up studies). See http://webdocs.cs.ualberta.ca/~sutton/RL-FAQ.html#behaviorism for further details of these research areas above.

## Literature Edit

### Conferences, journals Edit

Most reinforcement learning papers are published at the major machine learning and AI conferences (ICML, NIPS, AAAI, IJCAI, UAI, AI and Statistics) and journals (JAIR, JMLR, Machine learning journal). Some theory papers are published at COLT and ALT. However, many papers appear in robotics conferences (IROS, ICRA) and the "agent" conference AAMAS. Operations researchers publish their papers at the INFORMS conference and, for example, in the Operation Research, and the Mathematics of Operations Research journals. Control researchers publish their papers at the CDC and ACC conferences, or, e.g., in the journals IEEE Transactions on Automatic Control, or Automatica, although applied works tend to be published in more specialized journals. The Winter Simulation Conference also publishes many relevant papers. Other than this, papers also published in the major conferences of the neural networks, fuzzy, and evolutionary computation communities. The annual IEEE symposium titled Approximate Dynamic Programming and Reinforcement Learning (ADPRL) and the biannual European Workshop on Reinforcement Learning (EWRL) are two regularly held meetings where RL researchers meet.

## See also Edit

## ImplementationsEdit

- RL-Glue provides a standard interface that allows you to connect agents, environments, and experiment programs together, even if they are written in different languages.

- Maja Machine Learning Framework The Maja Machine Learning Framework (MMLF) is a general framework for problems in the domain of Reinforcement Learning (RL) written in python.

- TeachingBox is a Java reinforcement learning framework supporting many features like RBF networks, gradient descent learning methods, ...

- C++ and Python implementations for some well known reinforcement learning algorithms with source.

- Orange, a free data mining software suite, module orngReinforcement

- Policy Gradient Toolbox provides a package for learning about policy gradient approaches.

## References Edit

## External links Edit

- Website for
*Reinforcement Learning: An Introduction*(1998), by Rich Sutton and Andrew Barto, MIT Press, including a link to an html version of the book.

- Reinforcement Learning and Artificial Intelligence (RLAI, Rich Sutton's lab at the University of Alberta)

- Autonomous Learning Laboratory (ALL, Andrew Barto's lab at the University of Massachusetts Amherst)

- Relative Entropy Policy Searchement learning may be used to explain how equilibrium may arise under bounded rationality.

In machine learning, the environment is typically formulated as a Markov decision process (MDP), and many reinforcement learning algorithms for this context are highly related to dynamic programming techniques. The main difference between the classical techniques and reinforcement learning algorithms is that the latter do not need knowledge about the MDP and they target large MDPs where exact methods become infeasible.

Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge). The exploration vs. exploitation trade-off in reinforcement learning has been most thoroughly studied through the multi-armed bandit problem and in finite MDPs.

## Introduction Edit

The basic reinforcement learning model consists of:

- a set of environment states $ S $;

- a set of actions $ A $;

- rules of transitioning between states;

- rules that determine the
*scalar immediate reward*of a transition; and

- rules that describe what the agent observes.

The rules are often stochastic. The observation typically involves the scalar immediate reward associated with the last transition.

In many works, the agent is also assumed to observe the current environmental state, in which case we talk about *full observability*, whereas in the opposing case we talk about *partial observability*. Sometimes the set of actions available to the agent is restricted (e.g., you cannot spend more money than what you possess).

A reinforcement learning agent interacts with its environment in discrete time steps.

At each time $ t $, the agent receives an observation $ o_t $, which typically includes the reward $ r_t $.

It then chooses an action $ a_t $ from the set of actions available, which is subsequently sent to the environment.

The environment moves to a new state $ s_{t+1} $ and the reward $ r_{t+1} $ associated with the *transition* $ (s_t,a_t,s_{t+1}) $ is determined.

The goal of a reinforcement learning agent is to collect as much reward as possible. The agent can choose any action as a function of the history and it can even randomize its action selection.

When the agent's performance is compared to that of an agent which acts optimally from the beginning, the difference in performance gives rise to the notion of *regret*.

Note that in order to act near optimally, the agent must reason about the long term consequences of its actions: In order to maximize my future income I had better go to school now, although the immediate monetary reward associated with this might be negative.

Thus, reinforcement learning is particularly well suited to problems which include a long-term versus short-term reward trade-off. It has been applied successfully to various problems, including robot control, elevator scheduling, telecommunications, backgammon and checkers (Sutton and Barto 1998, Chapter 11).

Two components make reinforcement learning powerful:

The use of samples to optimize performance and the use of function approximation to deal with large environments.

Thanks to these two key components, reinforcement learning can be used in large environments in any of the following situations:

- A model of the environment is known, but an analytic solution is not available;

- Only a simulation model of the environment is given (the subject of simulation-based optimization);

- The only way to collect information about the environment is by interacting with it.

The first two of these problems could be considered planning problems (since some form of the model is available), while the last one could be considered as a genuine learning problem. However, under a reinforcement learning methodology both planning problems would be converted to machine learning problems.

## Exploration Edit

The reinforcement learning problem as described requires clever exploration mechanisms. Randomly selecting actions, without reference to an estimated probability distribution, is known to give rise to very poor performance. The case of (small) finite MDPs is relatively well understood by now.

However, due to the lack of algorithms that would provably scale well with the number of states (or scale to problems with infinite state spaces), in practice people resort to simple exploration methods. One such method is $ \epsilon $-greedy, when the agent chooses the action that it believes has the best long-term effect with probability $ 1-\epsilon $, and it chooses an action uniformly at random, otherwise. Here, $ 0<\epsilon<1 $ is a tuning parameter, which is sometimes changed, either according to a fixed schedule (making the agent explore less as time goes by), or adaptively based on some heuristics (Tokic & Palm, 2011).

## Algorithms for control learning Edit

Even if the issue of exploration is disregarded and even if the state was observable (which we assume from now on), the problem remains to find out which actions are good based on past experience.

### Criterion of optimality Edit

For simplicity, assume for a moment that the problem studied is *episodic*, an episode ending when some *terminal state* is reached. Assume further that no matter what course of actions the agent takes, termination is inevitable. Under some additional mild regularity conditions the expectation of the total reward is then well-defined, for *any* policy and any initial distribution over the states. Here, a policy refers to a mapping that assigns some probability distribution over the actions to all possible histories.

Given a fixed initial distribution $ \mu $, we can thus assign the expected return $ \rho^\pi $ to policy $ \pi $:

- $ \rho^\pi = E[R|\pi], $

where the random variable $ R $ denotes the *return* and is defined by

- $ R=\sum_{t=0}^{N-1} r_{t+1}, $

where $ r_{t+1} $ is the reward received after the $ t $-th transition, the initial state is sampled at random from $ \mu $ and actions are selected by policy $ \pi $. Here, $ N $ denotes the (random) time when a terminal state is reached, i.e., the time when the episode terminates.

In the case of non-episodic problems the return is often *discounted*,

- $ R=\sum_{t=0}^\infty \gamma^t r_{t+1}, $

giving rise to the total expected discounted reward criterion. Here $ 0 \le \gamma \le 1 $ is the so-called *discount-factor*. Since the undiscounted return is a special case of the discounted return, from now on we will assume discounting. Although this looks innocent enough, discounting is in fact problematic if one cares about online performance. This is because discounting makes the initial time steps more important. Since a learning agent is likely to make mistakes during the first few steps after its "life" starts, no uninformed learning algorithm can achieve near-optimal performance under discounting even if the class of environments is restricted to that of finite MDPs. (This does not mean though that, given enough time, a learning agent cannot figure how to act near-optimally, if time was restarted.)

The problem then is to specify an algorithm that can be used to find a policy with maximum expected return.

From the theory of MDPs it is known that, without loss of generality, the search can be restricted to the set of the so-called *stationary* policies. A policy is called stationary if the action-distribution returned by it depends only on the last state visited (which is part of the observation history of the agent, by our simplifying assumption). In fact, the search can be further restricted to *deterministic* stationary policies. A deterministic stationary policy is one which deterministically selects actions based on the current state. Since any such policy can be identified with a mapping from the set of states to the set of actions, these policies can be identified with such mappings with no loss of generality.

### Brute force Edit

The brute force approach entails the following two steps:

- For each possible policy, sample returns while following it

- Choose the policy with the largest expected return

One problem with this is that the number of policies can be extremely large, or even infinite. Another is that variance of the returns might be large, in which case a large number of samples will be required to accurately estimate the return of each policy.

These problems can be ameliorated if we assume some structure and perhaps allow samples generated from one policy to influence the estimates made for another. The two main approaches for achieving this are value function estimation and direct policy search.

### Value function approaches Edit

Value function approaches attempt to find a policy that maximizes the return by maintaining a set of estimates of expected returns for some policy (usually either the "current" or the optimal one).

These methods rely on the theory of MDPs, where optimality is defined in a sense which is stronger than the above one: A policy is called optimal if it achieves the best expected return from *any* initial state (i.e., initial distributions play no role in this definition). Again, one can always find an optimal policy amongst stationary policies.

To define optimality in a formal manner, define the value of a policy $ \pi $ by

- $ V^{\pi} (s) = E[R|s,\pi], $

where $ R $ stands for the random return associated with following $ \pi $ from the initial state $ s $.

Define $ V^*(s) $ as the maximum possible value of $ V^\pi(s) $, where $ \pi $ is allowed to change:

- $ V^*(s) = \sup \limits_\pi V^{\pi}(s). $

A policy which achieves these *optimal values* in *each* state is called *optimal*. Clearly, a policy optimal in this strong sense is also optimal in the sense that it maximizes the expected return $ \rho^\pi $, since $ \rho^\pi = E[ V^\pi(S) ] $, where $ S $ is a state randomly sampled from the distribution $ \mu $.

Although state-values suffice to define optimality, it will prove to be useful to define action-values. Given a state $ s $, an action $ a $ and a policy $ \pi $, the action-value of the pair $ (s,a) $ under $ \pi $ is defined by

- $ Q^\pi(s,a) = E[R|s,a,\pi],\, $

where, now, $ R $ stands for the random return associated with first taking action $ a $ in state $ s $ and following $ \pi $, thereafter.

It is well-known from the theory of MDPs that if someone gives us $ Q $ for an optimal policy, we can always choose optimal actions (and thus act optimally) by simply choosing the action with the highest value at each state.

The *action-value function* of such an optimal policy is called the *optimal action-value function* and is denoted by $ Q^* $.

In summary, the knowledge of the optimal action-value function *alone* suffices to know how to act optimally.

Assuming full knowledge of the MDP, there are two basic approaches to compute the optimal action-value function, value iteration and policy iteration.

Both algorithms compute a sequence of functions $ Q_k $ ($ k=0,1,2,\ldots, $) which converge to $ Q^* $.

Computing these functions involves computing expectations over the whole state-space, which is impractical for all, but the smallest (finite) MDPs, never mind the case when the MDP is unknown.

In reinforcement learning methods the expectations are approximated by averaging over samples and one uses function approximation techniques to cope with the need to represent value functions over large state-action spaces.

#### Monte Carlo methodsEdit

The simplest Monte Carlo methods can be used in an algorithm that mimics policy iteration.

Policy iteration consists of two steps: *policy evaluation* and *policy improvement*.

The Monte Carlo methods are used in the policy evaluation step.

In this step, given a stationary, deterministic policy $ \pi $, the goal is to compute the function values $ Q^\pi(s,a) $ (or a good approximation to them) for all state-action pairs $ (s,a) $.

Assume (for simplicity) that the MDP is finite and in fact a table representing the action-values fits into the memory.

Further, assume that the problem is episodic and after each episode a new one starts from some random initial state.

Then, the estimate of the value of a given state-action pair $ (s,a) $can be computed by simply averaging the sampled returns which originated from $ (s,a) $ over time.

Given enough time, this procedure can thus construct a precise estimate $ Q $ of the action-value function $ Q^\pi $.

This finishes the description of the policy evaluation step.

In the policy improvement step, as it is done in the standard policy iteration algorithm, the next policy is obtained by computing a *greedy* policy with respect to $ Q $: Given a state $ s $, this new policy returns an action that maximizes $ Q(s,\cdot) $. In practice one often avoids computing and storing the new policy, but uses lazy evaluation to defer the computation of the maximizing actions to when they are actually needed.

A few problems with this procedure are as follows:

- The procedure may waste too much time on evaluating a suboptimal policy;

- It uses samples inefficiently in that a long trajectory is used to improve the estimate only of the
*single*state-action pair that started the trajectory;

- When the returns along the trajectories have
*high variance*, convergence will be slow;

- It works in
*episodic problems only*;

- It works in
*small, finite MDPs only*.

#### Temporal difference methodsEdit

The first issue is easily corrected by allowing the procedure to change the policy (at all, or at some states) before the values settle. However good this sounds, this may be dangerous as this might prevent convergence. Still, most current algorithms implement this idea, giving rise to the class of *generalized policy iteration* algorithm. We note in passing that actor critic methods belong to this category.

The second issue can be corrected within the algorithm by allowing trajectories to contribute to any state-action pair in them.

This may also help to some extent with the third problem, although a better solution when returns have high variance is to use Sutton's temporal difference (TD) methods which are based on the recursive Bellman equation. Note that the computation in TD methods can be incremental (when after each transition the memory is changed and the transition is thrown away), or batch (when the transitions are collected and then the estimates are computed once based on a large number of transitions). Batch methods, a prime example of which is the least-squares temporal difference method due to Bradtke and Barto (1996), may use the information in the samples better, whereas incremental methods are the only choice when batch methods become infeasible due to their high computational or memory complexity. In addition, there exist methods that try to unify the advantages of the two approaches. Methods based on temporal differences also overcome the second but last issue.

In order to address the last issue mentioned in the previous section, *function approximation methods* are used.

In *linear function approximation* one starts with a mapping $ \phi $ that assigns a finite-dimensional vector to each state-action pair. Then, the action values of a state-action pair $ (s,a) $ are obtained by linearly combining the components of $ \phi(s,a) $ with some *weights* $ \theta $:

- $ Q(s,a) = \sum \limits_{i=1}^d \theta_i \phi_i(s,a) $.

The algorithms then adjust the weights, instead of adjusting the values associated with the individual state-action pairs.

However, linear function approximation is not the only choice.

More recently, methods based on ideas from nonparametric statistics (which can be seen to construct their own features) have been explored.

So far, the discussion was restricted to how policy iteration can be used as a basis of the designing reinforcement learning algorithms. Equally importantly, value iteration can also be used as a starting point, giving rise to the Q-Learning algorithm (Watkins 1989) and its many variants.

The problem with methods that use action-values is that they may need highly precise estimates of the competing action values, which can be hard to obtain when the returns are noisy. Though this problem is mitigated to some extent by temporal difference methods and if one uses the so-called compatible function approximation method, more work remains to be done to increase generality and efficiency. Another problem specific to temporal difference methods comes from their reliance on the recursive Bellman equation. Most temporal difference methods have a so-called $ \lambda $ parameter $ (0\le \lambda\le 1) $ that allows one to continuously interpolate between Monte-Carlo methods (which do not rely on the Bellman equations) and the basic temporal difference methods (which rely entirely on the Bellman equations), which can thus be effective in palliating this issue.

### Direct policy search Edit

An alternative method to find a good policy is to search directly in (some subset) of the policy space, in which case the problem becomes an instance of stochastic optimization. The two approaches available are gradient-based and gradient-free methods.

Gradient-based methods (giving rise to the so-called *policy gradient methods*) start with a mapping from a finite-dimensional (parameter) space to the space of policies: given the parameter vector $ \theta $, let $ \pi_\theta $ denote the policy associated to $ \theta $.

Define the performance function by

- $ \rho(\theta) = \rho^{\pi_\theta}. $

Under mild conditions this function will be differentiable as a function of the parameter vector $ \theta $.

If the gradient of $ \rho $ was known, one could use gradient ascent.

Since an analytic expression for the gradient is not available, one must rely on a noisy estimate.

Such an estimate can be constructed in many ways, giving rise to algorithms like Williams' REINFORCE method (which is also known as the likelihood ratio method in the simulation-based optimization literature).

Policy gradient methods have received a lot of attention in the last couple of years (e.g., Peters et al. (2003)), but they remain an active field.

The issue with many of these methods is that they may get stuck in local optima (as they are based on local search).

A large class of methods avoids relying on gradient information.

These include simulated annealing, cross-entropy search or methods of evolutionary computation.

Many gradient-free methods can achieve (in theory and in the limit) a global optimum.

In a number of cases they have indeed demonstrated remarkable performance.

The issue with policy search methods is that they may converge slowly if the information based on which they act is noisy.

For example, this happens when in episodic problems the trajectories are long and the variance of the returns is large. As argued beforehand, value-function based methods that rely on temporal differences might help in this case. In recent years, several actor-critic algorithms have been proposed following this idea and were demonstrated to perform well in various problems.

## Theory Edit

The theory for small, finite MDPs is quite mature.

Both the asymptotic and finite-sample behavior of most algorithms is well-understood.

As mentioned beforehand, algorithms with provably good online performance (addressing the exploration issue) are known.

The theory of large MDPs needs more work. Efficient exploration is largely untouched (except for the case of bandit problems).

Although finite-time performance bounds appeared for many algorithms in the recent years, these bounds are expected to be rather loose and thus more work is needed to better understand the relative advantages, as well as the limitations of these algorithms.

For incremental algorithm asymptotic convergence issues have been settled. Recently, new incremental, temporal-difference-based algorithms have appeared which converge under a much wider set of conditions than was previously possible (for example, when used with arbitrary, smooth function approximation).

## Current research Edit

Current research topics include:

adaptive methods which work with fewer (or no) parameters under a large number of conditions,

addressing the exploration problem in large MDPs,

large-scale empirical evaluations,

learning and acting under partial information (e.g., using Predictive State Representation),

modular and hierarchical reinforcement learning,

improving existing value-function and policy search methods,

algorithms that work well with large (or continuous) action spaces,

transfer learning,

lifelong learning,

efficient sample-based planning (e.g., based on Monte-Carlo tree search).

Multiagent or Distributed Reinforcement Learning is also a topic of interest in current research.

There is also a growing interest in real life applications of reinforcement learning.

Successes of reinforcement learning are collected on

here and

here.

Reinforcement learning algorithms such as TD learning are also being investigated as a model for Dopamine-based learning in the brain. In this model, the dopaminergic projections from the substantia nigra to the basal ganglia function as the prediction error. Reinforcement learning has also been used as a part of the model for human skill learning, especially in relation to the interaction between implicit and explicit learning in skill acquisition (the first publication on this application was in 1995-1996, and there have been many follow-up studies). See http://webdocs.cs.ualberta.ca/~sutton/RL-FAQ.html#behaviorism for further details of these research areas above.

## Literature Edit

### Conferences, journals Edit

Most reinforcement learning papers are published at the major machine learning and AI conferences (ICML, NIPS, AAAI, IJCAI, UAI, AI and Statistics) and journals (JAIR, JMLR, Machine learning journal). Some theory papers are published at COLT and ALT. However, many papers appear in robotics conferences (IROS, ICRA) and the "agent" conference AAMAS. Operations researchers publish their papers at the INFORMS conference and, for example, in the Operation Research, and the Mathematics of Operations Research journals. Control researchers publish their papers at the CDC and ACC conferences, or, e.g., in the journals IEEE Transactions on Automatic Control, or Automatica, although applied works tend to be published in more specialized journals. The Winter Simulation Conference also publishes many relevant papers. Other than this, papers also published in the major conferences of the neural networks, fuzzy, and evolutionary computation communities. The annual IEEE symposium titled Approximate Dynamic Programming and Reinforcement Learning (ADPRL) and the biannual European Workshop on Reinforcement Learning (EWRL) are two regularly held meetings where RL researchers meet.

## See also Edit

## ImplementationsEdit

- RL-Glue provides a standard interface that allows you to connect agents, environments, and experiment programs together, even if they are written in different languages.

- Maja Machine Learning Framework The Maja Machine Learning Framework (MMLF) is a general framework for problems in the domain of Reinforcement Learning (RL) written in python.

- TeachingBox is a Java reinforcement learning framework supporting many features like RBF networks, gradient descent learning methods, ...

- C++ and Python implementations for some well known reinforcement learning algorithms with source.

- Orange, a free data mining software suite, module orngReinforcement

- Policy Gradient Toolbox provides a package for learning about policy gradient approaches.

## References Edit

## External links Edit

- Website for
*Reinforcement Learning: An Introduction*(1998), by Rich Sutton and Andrew Barto, MIT Press, including a link to an html version of the book.

- Reinforcement Learning and Artificial Intelligence (RLAI, Rich Sutton's lab at the University of Alberta)

- Autonomous Learning Laboratory (ALL, Andrew Barto's lab at the University of Massachusetts Amherst)