Skip to main content
archive
Search Submit Donate Log in
Press Enter to search · Advanced search

Computer Science > Machine Learning

arXiv:2305.16554 (cs)
[Submitted on 26 May 2023]

Title:Emergent Agentic Transformer from Chain of Hindsight Experience

Authors:Hao Liu, Pieter Abbeel
View a PDF of the paper titled Emergent Agentic Transformer from Chain of Hindsight Experience, by Hao Liu and Pieter Abbeel
View PDF HTML (experimental)
Abstract:Large transformer models powered by diverse data and model scale have dominated natural language modeling and computer vision and pushed the frontier of multiple AI areas. In reinforcement learning (RL), despite many efforts into transformer-based policies, a key limitation, however, is that current transformer-based policies cannot learn by directly combining information from multiple sub-optimal trials. In this work, we address this issue using recently proposed chain of hindsight to relabel experience, where we train a transformer on a sequence of trajectory experience ascending sorted according to their total rewards. Our method consists of relabelling target return of each trajectory to the maximum total reward among in sequence of trajectories and training an autoregressive model to predict actions conditioning on past states, actions, rewards, target returns, and task completion tokens, the resulting model, Agentic Transformer (AT), can learn to improve upon itself both at training and test time. As we show on D4RL and ExoRL benchmarks, to the best our knowledge, this is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches, even from sub-optimal data. Our Agentic Transformer also shows a promising scaling trend that bigger models consistently improve results.
Comments: International Conference on Machine Learning (ICML) 2023
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2305.16554 [cs.LG]
  (or arXiv:2305.16554v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2305.16554
arXiv-issued DOI via DataCite

Submission history

From: Hao Liu [view email]
[v1] Fri, 26 May 2023 00:43:02 UTC (256 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Emergent Agentic Transformer from Chain of Hindsight Experience, by Hao Liu and Pieter Abbeel
  • View PDF
  • HTML (experimental)
  • TeX Source
view license

Current browse context:

cs.LG
< prev   |   next >
new | recent | 2023-05
Change to browse by:
cs

References & Citations

  • NASA ADS
  • Google Scholar
  • Semantic Scholar
Loading...

BibTeX formatted citation

Data provided by:

Bookmark

BibSonomy Reddit

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)

Code, Data and Media Associated with this Article

alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)

Demos

Replicate (What is Replicate?)
Hugging Face Spaces (What is Spaces?)
TXYZ.AI (What is TXYZ.AI?)

Recommenders and Search Tools

Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
IArxiv Recommender (What is IArxiv?)
  • Author
  • Venue
  • Institution
  • Topic

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
We gratefully acknowledge support from our major funders, member institutions, , and all contributors.
About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab)
Major funding support from
Simons Foundation Simons Foundation International Schmidt Sciences