Do my trips to the mound count against the pitcher?

Yes. A coach's visit counts the same as the manager's. In Majors the team gets one visit per inning and two per game; the next visit means the player must be removed as pitcher.

Any time out you're granted to talk to a defensive player counts as a visit to the pitcher, even if you never speak to the pitcher. Agree with your manager beforehand who goes out, and when.