Some post-compo work would go a long way to fleshing the idea out. It's not a bad premise, but the lack of feedback, scoring, and a clear goal, makes it a bit ambiguous and generally not interesting.
Now that LD is over, play around a bit and see what comes of it.
But overall it's a cool game you made