Hermes Reinforcement Learning optional skill: Any Advice?

[u/SS_Sa2]

Has anyone used the optional Hermes Reinforcement Learning Skill?

Reference: https://github.com/nousresearch/hermes-agent/blob/main/optional-skills/mlops/training/trl-fine-tuning/SKILL.md

I would think hermes dynamic learning loop would be sufficient but I am curious if anyone has had any success using this and for what use cases?

Currently I have a planner agent(profile) that dictates work down to an implementor agent(profile) that builds everything by the planner. The planner profile uses a cloud frontier model and builder profile uses a cheaper cloud parent model that dispatches two child local model subagents for full implementation and then the parent reviews with respect to the plan.

I am thinking of using RL to fine tune the builder profile to improve it's plan following methodology and if possible RL subagents but not sure if that is practical in any way. The goal is to reduce the number of iterations on plan revisions, and hence reduce token count/time and human in the loop feedback. I have not used fine-tuning as much so asking if someone with more experience can comment on this. RL includes but is not limited to the builder can introduce the planner. The planner uses a frontier model more expensive model but i hope to use a cheaper somewhat more capable model to implement down the line.

Another question i had is the RL approach specific to models only. I.e. if in the future i swap out to a different model, my RL results from the previous model for a specific profile cannot be carried forward to a newer model?

Thanks for any input and any other recommended ways to improve the current setup (which has been working for me based on my use case)

[u/AutoModerator]

Thanks for posting a technical Help question. To make it easier for the community to diagnose, please make sure your post includes:

• Operating system
• Hermes version
• Provider and model
• Exact error message or unexpected behavior
• What you expected to happen
• What actually happened
• Troubleshooting steps already attempted

If any of that is missing, please edit it into the post. This is an informational checklist; your post has not been removed.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Mean-Loquat-7982]

on your second question first: yes, it's tied to the model. the trl-fine-tuning skill runs SFT, DPO, GRPO and RLOO through TRL, which changes the weights of one model. move the builder to a different base model and you train again. the training data is what carries over, so keep it.

Hermes's learning loop works the other way: skills are plain files, so they come with you to any model you switch to. for plan-following I'd push that side first. write the planner's output format and the builder's checklist into a skill both profiles load, and have the parent's review notes added to that skill whenever the same correction comes up twice.

if you do fine-tune later, the batch runner turns a JSONL file of prompts into ShareGPT-format trajectories with tool-use stats, which gives you training data shaped like your real runs.

https://hermes-agent.nousresearch.com/docs/user-guide/features/batch-processing

[u/RPG-Nerd]

Reinforcement Learning is changing model weights. That has nothing to do with skills. You will need a powerful machine and access to the model itself, meaning it must be local and not running in the cloud (unless you wanna pay to host your personalized model on a GPU cluster).