Caterpillars is a good answer. Look at the undersides of the leaves for them, they can be very small and blend in with the color of the leaves.
ALSO, do you have grasshoppers? They eat the hell out of leaves of all kinds of plants. You will see ragged edged holes and chew areas similar to what you have there. They are a major PITA here and hide very well when you approach. Again, check under the leaves and all around the stems. They will skirt around that stem as they see you. They come in all kinds of colors too.
On one of my outdoor clones: