TD Lambda is an advanced algorithm used in reinforcement learning. It's an extension of other reinforcement learning algorithms, but it includes something called an eligibility trace.
What is an Eligibility Trace?
When using the TD Lambda algorithm, a vector called the eligibility trace keeps track of recent state valuations. The eligibility trace vector starts at zero and is incremented on each time step by the value gradient. Then it fades away over time by a particular factor. This eligibility trace vector helps keep track of which weight vector components contribute to recent state valuations.
How Does TD Lambda Work?
In TD Lambda, the weight vector is updated on each step proportional to the scalar TD error and the eligibility trace vector. The TD error represents the difference between reward received from an action and the projected reward from taking that action, for the current state. The weight vector is adjusted so that it more accurately reflects the true valuation of each state.
The update is done by applying the following formula:
new weight vector = old weight vector + learning rate * TD error * eligibility trace vector
Benefits of TD Lambda
TD Lambda has several benefits over other reinforcement learning algorithms. One major benefit is that it allows for faster learning. By including the eligibility trace vector, TD Lambda enables an agent to learn quicker because it better accounts for how recent state valuations are affected by previous state valuations. In addition, the ability to use a factor that fades over time also helps ensure that outdated information does not skew the valuation of a particular state.
TD Lambda offers a more advanced form of reinforcement learning. It includes eligibility trace which keeps track of recent state valuations for more accurate results. TD Lambda allows for faster learning and has other benefits as well. This algorithm is crucial for fields such as robotics and game design.
