Why hasn't my Handicap Index gone up much after some bad rounds?

Because of the caps (Rule 5.8 of the Rules of Handicapping). Each time your Handicap Index is updated, the new figure is compared with your Low Handicap Index: the lowest Handicap Index you've had in the 365 days before your most recent score. A Low Handicap Index applies once you have at least 20 acceptable scores.

There's no limit on how fast it can come down.

Your Handicap Committee can override the caps in circumstances such as injury. Committees are also expected to carry out a handicap review for every player at least once a year, and can adjust a Handicap Index that no longer reflects your ability.