Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), has emerged as a promising approach to fine-tuning large language models(LLMs) while reducing computational and memory overhead. However, LoRA assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across different modules and layers. AdaLoRA leverages Singular Value Decomposition (SVD) to parameterize updates and employs pruning of singular values to introduce dynamic rank allocation, thereby enhancing adaptability. However, during the training process, it often encounters issues of slow convergence speed and high computational overhead. To address this issue, we propose HyperAdaLoRA, a novel framework that accelerates the convergence of AdaLoRA by leveraging a hypernetwork. Instead of directly optimizing the components of Singular Value Decomposition (P, 𝛬, Q), HyperAdaLoRA employs a hypernetwork based on attention mechanisms to dynamically generate these parameters. By pruning the outputs of the hypernetwork that generates the singular values, dynamic rank allocation is achieved. Comprehensive experiments on various datasets and models demonstrate that our method achieves faster convergence without sacrificing performance. Moreover, our method generalizes well to other LoRA-based approaches, highlighting its strong generalization capability.

Prompt optimizers are widely used to create high-quality prompts for Large Language Models (LLMs), but their effectiveness remains unstable in practice. This instability is caused by the misalignment between conservative needs (e.g., safety compliance) and open-ended goals (e.g., creative writing). To address this, we propose a semantic-entropy-based method, using task uncertainty to guide prompt optimization. Specifically, we measure the task’s uncertainty level with pre-defined templates, then use this measure to direct prompt optimization: selecting high-entropy prompt candidates for creative tasks and low-entropy candidates for conservative ones. Extensive experiments across various model families demonstrate that our method consistently outperforms baselines by effectively adjusting entropy levels. Our approach requires no training, works with black-box models, and integrates easily into existing prompt optimizers.