---
title: japanese-stable-clip-vit-l-16
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/japanese-stable-clip-vit-l-16"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/japanese-stable-clip-vit-l-16.md"
repository: AI-ModelScope/japanese-stable-clip-vit-l-16
last_updated: 2023-11-24
license: other
pipeline_tag: feature-extraction
tasks:
  - feature-extraction
model_type:
  - japanese_stable_clip
architectures:
  - JapaneseStableCLIPModel
parameters: 414.4M
tensor_type:
  - F32
library_name:
  - pytorch
  - safetensors
frameworks:
  - pytorch
language:
  - ja
downloads: 68
stars: 0
tags:
  - clip
  - japanese-stable-clip
---

# japanese-stable-clip-vit-l-16

> japanese-stable-clip-vit-l-16 - AI-ModelScope 在 ModelScope 开源的模型。Japanese Stable CLIP ViT-L/16

AI-ModelScope/japanese-stable-clip-vit-l-16 是 ModelScope 魔搭社区上的 414.4M 参数feature-extraction模型，采用 other 许可。

- **Repository**: AI-ModelScope/japanese-stable-clip-vit-l-16
- **License**: other
- **Tasks**: feature-extraction
- **Parameters**: 414.4M
- **Tags**: clip, japanese-stable-clip
- **Downloads**: 68
- **Stars**: 0
- **Last updated**: 2023-11-24

Source: https://www.modelscope.cn/models/AI-ModelScope/japanese-stable-clip-vit-l-16

---

# Japanese Stable CLIP ViT-L/16

## Model Details

Japanese Stable CLIP is a Japanese [CLIP (Contrastive Language-Image Pre-Training)](https://arxiv.org/abs/2103.00020) model that enables to map both Japanese texts and images to the same embedding space. 
This model alone is capable of tasks such as zero-shot image classification and text-to-image retrieval. 
Furthermore, when combined with other components, it can be used as part of generative models, such as image-to-text and text-to-image generation.

## 示例代码
```python
from typing import Union, List
import ftfy, html, re, io
import requests
from PIL import Image
import torch
from modelscope import AutoModel, AutoTokenizer, AutoImageProcessor, BatchFeature

def basic_clean(text):
    text = ftfy.fix_text(text)
    text = html.unescape(html.unescape(text))
    return text.strip()

def whitespace_clean(text):
    text = re.sub(r"\s+", " ", text)
    text = text.strip()
    return text

def tokenize(
    tokenizer,
    texts: Union[str, List[str]],
    max_seq_len: int = 77,
):
    if isinstance(texts, str):
        texts = [texts]
    texts = [whitespace_clean(basic_clean(text)) for text in texts]

    inputs = tokenizer(
        texts,
        max_length=max_seq_len - 1,
        padding="max_length",
        truncation=True,
        add_special_tokens=False,
    )
    # add bos token at first place
    input_ids = [[tokenizer.bos_token_id] + ids for ids in inputs["input_ids"]]
    attention_mask = [[1] + am for am in inputs["attention_mask"]]
    position_ids = [list(range(0, len(input_ids[0])))] * len(texts)

    return BatchFeature(
        {
            "input_ids": torch.tensor(input_ids, dtype=torch.long),
            "attention_mask": torch.tensor(attention_mask, dtype=torch.long),
            "position_ids": torch.tensor(position_ids, dtype=torch.long),
        }
    )

device = "cuda" if torch.cuda.is_available() else "cpu"
model_path = "AI-ModelScope/japanese-stable-clip-vit-l-16"
model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_path)
processor = AutoImageProcessor.from_pretrained(model_path)

# Run!
image = Image.open(io.BytesIO(requests.get('https://images.pexels.com/photos/2253275/pexels-photo-2253275.jpeg?auto=compress&cs=tinysrgb&dpr=3&h=750&w=1260').content))
image = processor(images=image, return_tensors="pt").to(device)
text = tokenize(
    tokenizer=tokenizer,
    texts=["犬", "猫", "象"],
).to(device)

with torch.no_grad():
    image_features = model.get_image_features(**image)
    text_features = model.get_text_features(**text)
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs) 
```

## Usage

<details>

1. Install packages

  ```
  pip install ftfy pillow requests transformers torch sentencepiece protobuf
  ```

</details>


## Model Details

* **Developed by**: [Stability AI](https://stability.ai/)
* **Model type**: Contrastive Image-Text, Zero-Shot Image Classification 
* **Language(s)**: Japanese
* **License**: [STABILITY AI JAPANESE STABLE CLIP COMMUNITY LICENSE](./LICENSE).


| Model | ImageNet top-1 accuracy\* |
| :-- | --: |
| **Japanese Stable CLIP ViT-L/16** | 62.06 |
| [rinna/japanese-cloob-vit-b-16](https://huggingface.co/rinna/japanese-cloob-vit-b-16) | 54.64 |
| [laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k](https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k) | 53 |
| [rinna/japanese-clip-vit-b-16](https://huggingface.co/rinna/japanese-clip-vit-b-16) | 50.69 |

\* Computed scores based on https://github.com/rinnakk/japanese-clip. 


### Training


The model uses a ViT-L/16 Transformer architecture as an image encoder and a 12-layer BERT as a text encoder with the Japanese tokenizer from [rinna/japanese-roberta-base](https://huggingface.co/rinna/japanese-roberta-base).
During training, the image encoder was initialized from the [AugReg](https://arxiv.org/abs/2106.10270) [vit-large-patch16-224](https://huggingface.co/timm/vit_large_patch16_224.augreg_in21k_ft_in1k
) model and we applied [SigLIP (Sigmoid loss for Language-Image Pre-training)](https://arxiv.org/abs/2303.15343). 


### Training Dataset

The training dataset includes the following public datasets:

- [CC12M](https://github.com/google-research-datasets/conceptual-12m) with captions translated into Japanese
- [MS-COCO](https://cocodataset.org/#home) with [STAIR Captions](http://captions.stair.center/)


## Use and Limitations

### Intended Use

This model is intended to be used by the open-source community in vision-language applications.


### Limitations and bias


The training dataset may have contained offensive or inappropriate content even though we applied data filters. 
We recommend users exercise reasonable caution when using these models in production systems. Do not use the model for any applications that may cause harm or distress to individuals or groups.


## How to cite

```bibtex
@misc{JapaneseStableCLIP, 
    url    = {[https://huggingface.co/stabilityai/japanese-stable-clip-vit-l-16](https://huggingface.co/stabilityai/japanese-stable-clip-vit-l-16)}, 
    title  = {Japanese Stable CLIP ViT-L/16}, 
    author = {Shing, Makoto and Akiba, Takuya}
}
```

## Contact
* For questions and comments about the model, please join [Stable Community Japan](https://discord.com/invite/StableJP).
* For future announcements / information about Stability AI models, research, and events, please follow https://twitter.com/StabilityAI_JP.
* For business and partnership inquiries, please contact partners-jp@stability.ai. ビジネスや協業に関するお問い合わせはpartners-jp@stability.aiにご連絡ください。