Millimeter-wave (mmWave) communications are a key enabler of next-generation wireless networks, thanks to their abundant spectrum resources. Cluster-based models offer a compact and physically interpretable abstraction by grouping multipath components (MPCs) with similar properties, but their accuracy and adaptability are often limited by insufficient environmental information. To address this limitations, we propose a predictive channel modeling framework that jointly estimates MPCs clusters and associated parameters by leveraging multimodal environmental data, including light detection and ranging (LiDAR), images, and locations. The framework extracts modality-specific features and employs a hierarchical attention-based fusion architecture to integrate complementary information. Furthermore, a transfer learning strategy is introduced to support efficient model adaptation across different indoor regions. Experimental results demonstrate that multimodal fusion significantly outperforms single-modality baselines, which reduces a relative root mean squared error (RMSE) of RMS delay spread (DS) prediction by over 50% and improves cluster classification accuracy by around 6%. Meanwhile, the multi-task learning brings an additional about 5% gain in cluster prediction. Moreover, in inter-region transfer scenarios, the model achieves over 87% accuracy using only 20% data, highlighting strong data efficiency and adaptability.