OpenDuckMini快速入门教程-强化学习训练

文章说明

• 本教程介绍生成pkl文件后的训练步骤
• 本教程只是简单介绍下操作步骤，更多详细内容请参阅源码 | 源码地址
• 欢迎加入我们的openduckmini交流群，微信扫描右侧二维码立即进群交流

相关设备

• OpenDuckMini套件: 采购地址

操作步骤

• 复制polynomial_coefficients.pkl文件

$ cp ~/open_duck_mini_ws/Open_Duck_reference_motion_generator/polynomial_coefficients.pkl ~/open_duck_mini_ws/Open_Duck_Playground/playground/open_duck_mini_v2/data/

• 设置USE_IMITATION_REWARD=True

$ vim ~/open_duck_mini_ws/Open_Duck_Playground/playground/open_duck_mini_v2/joystick.py

• 进行训练

$ cd ~/open_duck_mini_ws/Open_Duck_Playground
$ uv run playground/open_duck_mini_v2/runner.py --task flat_terrain_backlash --num_timesteps 300000000

• 启动TensorBoard可视化工具，可视化训练过程和指标

$ uv run tensorboard --logdir=<yourlogdir>

• 最终的训练结果ONNX.onnx会生成在当前目录下

FAQ

训练中出现的错误大多是环境配置错误

问题一：报错AttributeError: 'dict' object has no attribute 'policy'

Traceback (most recent call last):
  File "/root/open_duck_mini_ws/Open_Duck_Playground/playground/open_duck_mini_v2/runner.py", line 64, in <module>
    main()
  File "/root/open_duck_mini_ws/Open_Duck_Playground/playground/open_duck_mini_v2/runner.py", line 60, in main
    runner.train()
  File "/root/open_duck_mini_ws/Open_Duck_Playground/playground/common/runner.py", line 114, in train
    _, params, _ = train_fn(
                   ^^^^^^^^^
  File "/root/open_duck_mini_ws/Open_Duck_Playground/.venv/lib/python3.12/site-packages/brax/training/agents/ppo/train.py", line 692, in train
    policy_params_fn(current_step, make_policy, params)
  File "/root/open_duck_mini_ws/Open_Duck_Playground/playground/common/runner.py", line 78, in policy_params_fn
    export_onnx(
  File "/root/open_duck_mini_ws/Open_Duck_Playground/playground/common/export_onnx.py", line 150, in export_onnx
    transfer_weights(params[1].policy["params"], tf_policy_network)
                     ^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'policy'

解决方法

• 将runner.py文件中的params[1].policy["params"] 改成params[1]["params"]，修改后能正常训练，但不确认时候会影响训练效果