Randomized smoothing (RS) has provided state-of-the-art (SOTA) certified robustness against adversarial perturbations for large neural networks. Among studies in this field, methods based on adversarial training (AT) achieve remarkably robust performance by applying adversarial examples to construct the smoothed classifier. These AT-based RS methods typically seek a pointwise adversary that generates the worst-case adversarial examples by perturbing each input independently. However, there are unexplored benefits to considering such adversarial robustness across the entire data distribution. To this end, we provide a novel framework called DRF, which connects AT-based RS methods with distributional robustness (DR), and show that these methods are special cases of their counterparts in our framework. Due to the advantages conferred by DR, our framework can control the trade-off between the clean accuracy and certified robustness of smoothed classifiers to a significant extent. Our experiments demonstrate that DRF can substantially improve the certified robustness of AT-based RS.
@inproceedings{wang2024drf,title={DRF: Improving Certified Robustness via Distributional Robustness Framework},author={Wang, Zekai and Zhou, Zhengyu and Liu, Weiwei},booktitle={AAAI Conference on Artificial Intelligence},volume={38},number={14},pages={15752--15760},year={2024}}
Contrastive adversarial training has successfully improved the robustness of contrastive learning (CL). However, the robustness metric in these methods depends on attack algorithms, image labels, and downstream tasks, introducing reliability concerns. To address these issues, this paper proposes a novel Robustness Verification framework for Contrastive Learning (RVCL). Specifically, we define the verification problem of CL from deterministic and probabilistic perspectives, then provide several effective metrics to evaluate the robustness of CL encoder. Furthermore, we use extreme value theory to reveal the relationship between the robust radius of the CL encoder and that of the supervised downstream task. Extensive experiments on various benchmark models and datasets validate theoretical findings, and further demonstrate RVCL’s capability to evaluate the robustness of both CL encoders and images.
@article{wang2023rvcl,title={RVCL: Evaluating the Robustness of Contrastive Learning via Verification},author={Wang, Zekai and Liu, Weiwei},journal={Journal of Machine Learning Research},volume={24},number={396},pages={1--43},year={2023}}
It has been recognized that the data generated by the denoising diffusion probabilistic model (DDPM) improves adversarial training. After two years of rapid development in diffusion models, a question naturally arises: can better diffusion models further improve adversarial training? This paper gives an affirmative answer by employing the most recent diffusion model which has higher efficiency (∼20 sampling steps) and image quality (lower FID score) compared with DDPM. Our adversarially trained models achieve state-of-the-art performance on RobustBench using only generated data (no external datasets). Under the L∞-norm threat model with ε=8/255, our models achieve 70.69% and 42.67% robust accuracy on CIFAR-10 and CIFAR-100, respectively, i.e. improving upon previous state-of-the-art models by +4.58% and +8.03%. Under the L2-norm threat model with ε=128/255, our models achieve 84.86% on CIFAR-10 (+4.44%). These results also beat previous works that use external data. We also provide compelling results on the SVHN and TinyImageNet datasets.
@inproceedings{wang2023better,title={Better Diffusion Models Further Improve Adversarial Training},author={Wang, Zekai and Pang, Tianyu and Du, Chao and Lin, Min and Liu, Weiwei and Yan, Shuicheng},booktitle={International Conference on Machine Learning},volume={202},pages={36246--36263},year={2023}}
TKDE
Task Variance Regularized Multi-Task Learning
Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, and Wenbin Hu
IEEE Transactions on Knowledge and Data Engineering, 2023
Multi-task Learning (MTL), which involves the simultaneous learning of multiple tasks, can achieve better performance than learning each task independently. It has achieved great success in various applications, ranging from Computer Vision (CV) to Natural Language Processing (NLP). In MTL, the losses of the including tasks are jointly optimized. However, it is common for these tasks to be competing. When the tasks are competing, minimizing the losses of some tasks increases the losses of others, which accordingly increases the task variance (variance between the task-specific loss); furthermore, it induces under-fitting in some tasks and over-fitting in others, which degenerates the generalization performance of an MTL model. To address this issue, it is necessary to control the task variance; thus, task variance regularization is a natural choice. While intuitive, task variance regularization remains unexplored in MTL. Accordingly, to fill this gap, we study the generalization error bound of MTL through the lens of task variance and propose the task variance matters the generalization performance of MTL. Furthermore, this paper investigates how the task variance might be effectively regularized, and consequently proposes a multi-task learning method based on adversarial multi-armed bandit. The proposed method, dubbed BanditMTL, regularizes the task variance by means of a mirror gradient ascent-descent algorithm. Adopting BanditMTL both in CV and NLP applications is found to achieve state-of-the-art performance. The results of extensive experiments back up our theoretical analysis and validate the superiority of our proposals.
@article{mao2023task,title={Task Variance Regularized Multi-Task Learning},author={Mao, Yuren and Wang, Zekai and Liu, Weiwei and Lin, Xuemin and Hu, Wenbin},journal={IEEE Transactions on Knowledge and Data Engineering},volume={35},number={8},pages={8615-8629},year={2023}}
Contrastive adversarial training has successfully improved the robustness of contrastive learning (CL). However, the robustness metric used in these methods is linked to attack algorithms, image labels and downstream tasks, all of which may affect the consistency and reliability of robustness metric for CL. To address these problems, this paper proposes a novel Robustness Verification framework for Contrastive Learning (RVCL). Furthermore, we use extreme value theory to reveal the relationship between the robust radius of the CL encoder and that of the supervised downstream task. Extensive experimental results on various benchmark models and datasets verify our theoretical findings, and further demonstrate that our proposed RVCL is able to evaluate the robustness of both models and images.
@inproceedings{wang2022robustness,title={Robustness Verification for Contrastive Learning},author={Wang, Zekai and Liu, Weiwei},booktitle={International Conference on Machine Learning},volume={162},pages={22865--22883},year={2022}}
Interestingly, recent experimental results have identified a robust fairness phenomenon in adversarial training (AT), namely that a robust model well-trained by AT exhibits a remarkable disparity of standard accuracy and robust accuracy among different classes compared with natural training. However, the effect of different perturbation radii in AT on robust fairness has not been studied, and one natural question is raised: does a tradeoff exist between average robustness and robust fairness? Our extensive experimental results provide an affirmative answer to this question: with an increasing perturbation radius, stronger AT will lead to a larger class-wise disparity of robust accuracy. Theoretically, we analyze the class-wise performance of adversarially trained linear models with mixture Gaussian distribution. Our theoretical results support our observations. Moreover, our theory shows that adversarial training easily leads to more serious robust fairness issue than natural training. Motivated by theoretical results, we propose a fairly adversarial training (FAT) method to mitigate the tradeoff between average robustness and robust fairness. Experimental results validate the effectiveness of our proposed method.
@inproceedings{ma2022tradeoff,title={On the Tradeoff Between Robustness and Fairness},author={Ma, Xinsong and Wang, Zekai and Liu, Weiwei},booktitle={Advances in Neural Information Processing Systems},volume={35},pages={26230--26241},year={2022}}
Task weighting, which assigns weights on the including tasks during training, significantly matters the performance of Multi-task Learning (MTL); thus, recently, there has been an explosive interest in it. However, existing task weighting methods assign weights only based on the training loss, while ignoring the gap between the training loss and generalization loss. It degenerates MTL’s performance. To address this issue, the present paper proposes a novel task weighting algorithm, which automatically weights the tasks via a learning-to-learn paradigm, referred to as MetaWeighting. Extensive experiments are conducted to validate the superiority of our proposed method in multi-task text classification.
@inproceedings{mao2022metaweighting,title={MetaWeighting: Learning to Weight Tasks in Multi-Task Learning},author={Mao, Yuren and Wang, Zekai and Liu, Weiwei and Lin, Xuemin and Xie, Pengtao},booktitle={Findings of the Association for Computational Linguistics},pages={3436--3448},year={2022}}
Contrastive adversarial training has successfully improved the robustness of contrastive learning (CL). However, the robustness metric used in these methods is linked to attack algorithms, image labels and downstream tasks, all of which may affect the consistency and reliability of robustness metric for CL. To address these problems, this paper proposes a novel Robustness Verification framework for Contrastive Learning (RVCL). Furthermore, we use extreme value theory to reveal the relationship between the robust radius of the CL encoder and that of the supervised downstream task. Extensive experimental results on various benchmark models and datasets verify our theoretical findings, and further demonstrate that our proposed RVCL is able to evaluate the robustness of both models and images.
MEMS
Deep learning for non-parameterized MEMS structural design
Ruiqi Guo, Fanping Sui, Wei Yue, Zekai Wang, Sedat Pala, Kunying Li, Renxiao Xu, and Liwei Lin
The geometric designs of MEMS devices can profoundly impact their physical properties and eventual performances. However, it is challenging for researchers to rationally consider a large number of possible designs, as it would be very time- and resource-consuming to study all these cases using numerical simulation. In this paper, we report the use of deep learning techniques to accelerate the MEMS design cycle by quickly and accurately predicting the physical properties of numerous design candidates with vastly different geometric features. Design candidates are represented in a nonparameterized, topologically unconstrained form using pixelated black-and-white images. After sufficient training, a deep neural network can quickly calculate the physical properties of interest with good accuracy without using conventional numerical tools such as finite element analysis. As an example, we apply our deep learning approach in the prediction of the modal frequency and quality factor of disk-shaped microscale resonators. With reasonable training, our deep learning neural network becomes a high-speed, high-accuracy calculator: it can identify the flexural mode frequency and the quality factor 4.6 × 103 times and 2.6 × 104 times faster, respectively, than conventional numerical simulation packages, with good accuracies of 98.8 ± 1.6% and 96.8 ± 3.1%, respectively. When simultaneously predicting the frequency and the quality factor, up to ∼96.0% of the total computation time can be saved during the design process. The proposed technique can rapidly screen over thousands of design candidates and promotes experience-free and data-driven MEMS structural designs.
@article{guo2022deep,title={Deep learning for non-parameterized MEMS structural design},author={Guo, Ruiqi and Sui, Fanping and Yue, Wei and Wang, Zekai and Pala, Sedat and Li, Kunying and Xu, Renxiao and Lin, Liwei},journal={Microsystems \& Nanoengineering},volume={8},number={1},pages={91},year={2022},publisher={Nature Publishing Group UK London}}
Task variance regularization, which can be used to improve the generalization of Multi-task Learning (MTL) models, remains unexplored in multi-task text classification. Accordingly, to fill this gap, this paper investigates how the task might be effectively regularized, and consequently proposes a multi-task learning method based on adversarial multi-armed bandit. The proposed method, named BanditMTL, regularizes the task variance by means of a mirror gradient ascent-descent algorithm. Adopting BanditMTL in the multi-task text classification context is found to achieve state-of-the-art performance. The results of extensive experiments back up our theoretical analysis and validate the superiority of our proposals.
@inproceedings{mao2021banditmtl,title={BanditMTL: Bandit-based Multi-task Learning for Text Classification},author={Mao, Yuren and Wang, Zekai and Liu, Weiwei and Lin, Xuemin and Hu, Wenbin},booktitle={Annual Meeting of the Association for Computational Linguistics},pages={5506--5516},year={2021}}
MEMS
Accelerating MEMS design process through machine learning from pixelated binary images
Ruiqi Guo, Renxiao Xu, Zekai Wang, Fanping Sui, and Liwei Lin
In International Conference on Micro Electro Mechanical Systems, 2021
This paper reports the use of machine learning in accelerating the MEMS design process. Candidate designs are represented by pixelated binary 2D images. Instead of common computational tools like FEA, we use trained neural network for quickly obtaining physical properties of interest for each candidate design. Circular disk resonators are used as an example to demonstrate the capability of our method. After sufficient training with 9000 images, the resulting neural network can serve as a high-speed, high-accuracy analyzer: it can identify four vibrational modes of interest and calculate the corresponding frequencies 4000 times faster than commonly used FEA software, with remarkable accuracy (∼98%).
@inproceedings{guo2021accelerating,title={Accelerating MEMS design process through machine learning from pixelated binary images},author={Guo, Ruiqi and Xu, Renxiao and Wang, Zekai and Sui, Fanping and Lin, Liwei},booktitle={International Conference on Micro Electro Mechanical Systems},pages={153--156},year={2021}}