As I have included the number of distinct providers/vendors for a member in my models, the accuracy on my validation or out of bag sets dramatically increased (delta RMSLE of -0.02). However when I submitted these models, they performed very badly on the leaderboard data (delta RMSLE +0.02).
Normalizing these variables for each year didn't help to improve the discrepancy between validation and leaderboard data. 

Has anyone else observed a similar behaviour with these predictors? Is there an explanation for this or am I making a mistake?

"Distinct Prov/Vend" would be a predictor highly related to what type of care/what type of patients are in this Heritage 'Provider Network".

From their website, you can see Heritage quickly growing their "lives"and regional footprint in the last 5 years. One year, it almost doubles the number pateints.

If Y4 corresponds to one of these years, then training data would reflect somehwat outdated information as to the type of specialists available / type of patients under PCPs.