--- title: kaleido-xxl canonical_url: "https://www.modelscope.cn/models/LLM-Research/kaleido-xxl" md_url: "https://www.modelscope.cn/models/LLM-Research/kaleido-xxl.md" repository: LLM-Research/kaleido-xxl last_updated: 2024-09-09 license: apache-2.0 pipeline_tag: text2text-generation tasks: - text2text-generation model_type: - t5 architectures: - T5ForConditionalGeneration library_name: - transformer - pytorch frameworks: - pytorch downloads: 2079 stars: 0 tags: - generated_from_trainer --- # kaleido-xxl > kaleido-xxl - LLM-Research 在 ModelScope 开源的模型。Model Card for Kaleido LLM-Research/kaleido-xxl 是 ModelScope 魔搭社区上的text2text-generation模型，采用 apache-2.0 许可。 - **Repository**: LLM-Research/kaleido-xxl - **License**: apache-2.0 - **Tasks**: text2text-generation - **Tags**: generated_from_trainer - **Downloads**: 2079 - **Stars**: 0 - **Last updated**: 2024-09-09 Source: https://www.modelscope.cn/models/LLM-Research/kaleido-xxl --- # Model Card for Kaleido ## Model Description Kaleido is a multi-task seq2seq model designed to _generate_, _explain_, and output the _relevance_ and _valence_ of contextualized values, rights, and duties, distilled from GPT-4 generated data. - **Model type:** Language model - **Language(s) (NLP):** en - **License:** [AI2 ImpACT License - Medium Risk Artifacts (“MR Agreement”)](https://allenai.org/licenses/impact-mr) - **Parent Model:** [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl) - **GitHub Repo:** https://github.com/tsor13/kaleido - **Paper:** https://arxiv.org/abs/2309.00779 - **Demo:** https://kaleido.allen.ai - **All model sizes:** [small](https://huggingface.co/tsor13/kaleido-small), [base](https://huggingface.co/tsor13/kaleido-base), [large](https://huggingface.co/tsor13/kaleido-large), [xl](https://huggingface.co/tsor13/kaleido-xl), [xxl](https://huggingface.co/tsor13/kaleido-xxl) # Uses It is intended to be used for research purposes to a) understand how well large language models can approximate pluralistic human values and b) to make an open, transparent attempt to increase the capabilities of LLMs to model human values. ## Out-of-Scope Use The model is not intended to be used for advice, human-facing applications, or other purposes. # Bias, Risks, and Limitations Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. Additionally, certain groups may be represented better in the model's outputs than others, and the fact that the data is entirely in English and generated by predominantly by English speakers/LLMs trained on English, the model's outputs likely fit perspectives from English-speaking countries better. The relevance score _should not_ be interpreted as an importance score, but, due to the composition of the training data, corresponds more closely with "is this value likely to have been generated for this situation by GPT-4?" ## Recommendations We recommend that this model not be used for any high-impact or human-facing purposes as its biases and limitations need to be further explored. We intend this to be a research artifact to advance AI's ability to model and interact with pluralistic human values, rights, and duties. # Training Details ## Training Data The model is trained on the `mixture` training split of [ValuePrism](https://huggingface.co/datasets/tsor13/ValuePrism). ## Training Procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3e-05 - train_batch_size: 64 - eval_batch_size: 8 - seed: 42 - distributed_type: deepspeed - num_devices: 8 - total_train_batch_size: 64 - total_eval_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 4.0 ### Framework versions - Transformers 4.22.0.dev0 - Pytorch 1.12.1+cu113 - Datasets 2.4.0 - Tokenizers 0.12.1 ## Testing Data, Factors & Metrics ### Testing Data The model is tested on the four subtasks in [ValuePrism](https://huggingface.co/datasets/tsor13/ValuePrism). ### Metrics Accuracy is used for relevance and valence, as they are classification tasks, and perplexity is used for generation and explanation, as they are free text generation tasks. ## Results | **Model** | **Relevance Acc ↑** | **Valence Acc ↑** | **Generative Perp ↓** | **Explanation Perp ↓** | |-------------------------|------------------|-------------------|-----------------|------------------| | `kaleido-xxl` (11B) | 89.1 | 81.9 | 2.22 | 2.99 | | `kaleido-xl` (3B) | 88.4 | 80.8 | 2.23 | 3.14 | | `kaleido-large` (770M) | 87.2 | 79.2 | 2.34 | 3.52 | | `kaleido-base` (220M) | 83.5 | 74.5 | 2.53 | 4.23 | | `kaleido-small` (60M) | 66.0 | 59.7 | 2.86 | 5.70 | ## Model Architecture and Objective Kaleido is an encoder-decoder T5-based model trained using negative log-likelihood. # Citation **BibTeX:** ``` @misc{sorensen2023value, title={Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties}, author={Taylor Sorensen and Liwei Jiang and Jena Hwang and Sydney Levine and Valentina Pyatkin and Peter West and Nouha Dziri and Ximing Lu and Kavel Rao and Chandra Bhagavatula and Maarten Sap and John Tasioulas and Yejin Choi}, year={2023}, eprint={2309.00779}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` # Model Card Contact Contact Taylor Sorensen (`tsor13@cs.washington.edu`) for any questions about this model. # How to Get Started with the Model Use the code below to get started with the model. **Load the model:** ```python from transformers import AutoTokenizer, AutoModelForSeq2SeqLM model_name = 'allenai/kaleido-xxl' tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSeq2SeqLM.from_pretrained(model_name) ``` Each task ('generate', 'relevance', 'valence', 'explanation') has its own template that can be accessed from the model config. ```python task = 'generate' # can be 'generate', 'relevance', 'valence', or 'explanation' model.config.task_specific_params[task]['template'] ``` Output: ``` '[Generate]:\tAction: ACTION' ``` The generate template requires `ACTION`, while the other three templates require `ACTION`, `VRD` (`'Value', 'Right', or 'Duty'`) and `TEXT`. Replace the arguments with the text and generate with the model. Generate example: ```python action = 'Go to the gym' input_text = model.config.task_specific_params['generate']['template'].replace('ACTION', action) # tokenize input_ids = tokenizer.encode(input_text, return_tensors='pt') # generate output_ids = model.generate(input_ids, max_length=64) # decode output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True) output_text ``` Output: ``` 'Value: Personal growth' ``` Valence example: ```python action = 'Go to the gym' vrd = 'Value' text = 'Health' input_text = model.config.task_specific_params['valence']['template'] replacements = { 'ACTION': action, 'VRD': vrd, 'TEXT': text } for key, value in replacements.items(): input_text = input_text.replace(key, value) # tokenize input_ids = tokenizer.encode(input_text, return_tensors='pt') # generate output_ids = model.generate(input_ids, max_length=64) # decode output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True) output_text ``` Output: ``` 'Supports' ``` Alternatively, one could use the `KaleidoSys` class in the companion [git repo](https://github.com/tsor13/kaleido/tree/main) which automates templatization and such.