Reinforcement learning addresses the problem of learning actions in order to maximize an agent's performance. To scale reinforcement learning to complex real-world tasks, the agent must be able to discover hierarchical structures within their learning and control systems. This paper presents a method to discover subgoals with particular structural properties. By discovering subgoals and including policies to subgoals as actions, the agent can increase it's performance in other tasks. The agent discovers the subgoals by searching a learned policy model for state that exhibits certain structural properties. This approach is illustrated using gridworld tasks.