Amami will find an unsigned handwritten note waiting for him the next time he goes to tend to the wisteria tree -

I hope you'll show both of them a beautiful garden.

After he reads it, he will be able to touch the wisteria trees without consequence.