--- title: Hi3Dv1.0-mlx canonical_url: "https://www.modelscope.cn/models/mlx-community/Hi3Dv1.0-mlx" md_url: "https://www.modelscope.cn/models/mlx-community/Hi3Dv1.0-mlx.md" repository: mlx-community/Hi3Dv1.0-mlx last_updated: 2026-10-23 license: other pipeline_tag: image-to-3D tasks: - image-to-3D model_type: - hi3d base_model: - hi3d-official/Hi3Dv1.0 base_model_relation: finetune parameters: 4.3B tensor_type: - F32 - F16 library_name: - mlx - safetensors - pytorch frameworks: - pytorch downloads: 24 stars: 1 tags: - mlx - mlx-vlm - apple-silicon - image-to-3d - mesh - sparc3d --- # Hi3Dv1.0-mlx > Hi3Dv1.0-mlx - mlx-community 在 ModelScope 开源的模型。Install: until the hi3d model is in an mlx-vlm release, install it from source: pip install git+https://github.com/cguldogan/mlx-vlm.git@hi3d. mlx-community/Hi3Dv1.0-mlx 是 ModelScope 魔搭社区上的 4.3B 参数image-to-3D模型，采用 other 许可，基于 hi3d-official/Hi3Dv1.0 构建。 - **Repository**: mlx-community/Hi3Dv1.0-mlx - **License**: other - **Tasks**: image-to-3D - **Parameters**: 4.3B - **Base model**: hi3d-official/Hi3Dv1.0 - **Tags**: mlx, mlx-vlm, apple-silicon, image-to-3d, mesh, sparc3d - **Downloads**: 24 - **Stars**: 1 - **Last updated**: 2026-10-23 Source: https://www.modelscope.cn/models/mlx-community/Hi3Dv1.0-mlx --- # Hi3D v1.0 for MLX > **Install:** until the `hi3d` model is in an mlx-vlm release, install it from > source: `pip install git+https://github.com/cguldogan/mlx-vlm.git@hi3d`. Inference-only MLX port of [Hi3D v1.0](https://huggingface.co/hi3d-official/Hi3Dv1.0), an image-conditioned mesh refiner built on [Sparc3D](https://arxiv.org/abs/2505.14521). Hi3D takes an image of an object and a coarse mesh of the same object and returns high-resolution geometry (no texture). The reference implementation needs Linux, an NVIDIA GPU, spconv, flash-attn and Triton; this port runs on Apple silicon with MLX and NumPy. The [converted bundle](https://huggingface.co/mlx-community/Hi3Dv1.0-mlx) holds the Hi3D weights in their released dtypes together with the DINOv2-L and CLIP ViT-L/14 image encoders it conditions on. ## Usage ```sh hf download mlx-community/Hi3Dv1.0-mlx --local-dir models/Hi3Dv1.0-mlx ``` Refine an existing coarse mesh: ```sh python -m mlx_vlm.models.hi3d.generate \ --model models/Hi3Dv1.0-mlx \ --image object.png --mesh coarse.obj \ --output refined.obj --resolution 1024 ``` Or build the coarse mesh from the same image with [SAM 3D Objects](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/sam3d_objects/README.md), so the whole image-to-mesh path runs in MLX. The image's alpha channel (or `--mask`) selects the object: ```sh hf download mlx-community/sam-3d-objects-bf16 --local-dir models/sam-3d-objects-bf16 python -m mlx_vlm.models.hi3d.generate \ --model models/Hi3Dv1.0-mlx \ --sam3d-model models/sam-3d-objects-bf16 \ --image object.png --output refined.obj --resolution 1024 ``` From Python: ```python from PIL import Image from mlx_vlm.models.hi3d import Model from mlx_vlm.models.hi3d.mesh import read_mesh, write_obj model = Model.from_pretrained("models/Hi3Dv1.0-mlx") vertices, faces = read_mesh("coarse.obj") result = model(Image.open("object.png"), vertices, faces, resolution=1024, seed=0) write_obj("refined.obj", result.vertices, result.faces) ``` The model also loads through `mlx_vlm.utils.load_model`. ## Inputs and outputs - **Image:** an RGB or RGBA image. Transparent pixels mark the background; the subject is cropped to a square, composited on white and padded so it fills `1 / 1.2` of the frame, the framing the conditioner was trained on. - **Coarse mesh:** any triangle mesh of the same object. Hi3D re-voxelizes it, so position and scale do not matter, but orientation does. The reference demo feeds Hunyuan3D-2 meshes unchanged and documents no frame; this port places the side shown in the image facing +Z with +Y up, which `mesh.coarse_from_sam3d` does for a SAM 3D Objects result. If refined detail lands on the hidden side, turn the coarse mesh 180 degrees about Y. `read_mesh` reads OBJ files directly and other formats through the optional `trimesh` package. - **Resolution:** 512 (one pass), 1024 (two passes) or 1536 (three passes). 512 and 1024 are tested in this port; 1536 is not. Upstream validated it only on a 96 GB GPU. - **Result:** vertices centered at the origin with unit maximum radius, and triangle faces. Small disconnected pieces are removed. When the optional `pyfqmr` package is installed the mesh is simplified to the released face targets (1M faces at 512 and 1024, 2M at 1536); otherwise it is kept whole. ## How it works | Stage | Module | | --- | --- | | Frame the image; DINOv2-L tokens at 756 px joined with the CLIP-L embedding | [vision.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/vision.py) (reuses `dinov2` and the `llava` CLIP tower) | | Surface-voxelize the coarse mesh within an occupied-voxel budget | [voxelize.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/voxelize.py) (NumPy port of the Triton kernel) | | Heun flow sampling with token-warped schedule and late guidance | [pipeline.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/pipeline.py) | | Sparse latent flow transformer: sparse residual I/O blocks, 24 full-attention blocks with spatial RoPE | [transformer.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/transformer.py), [sparse.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/sparse.py) | | Point Transformer V3 decoder: serialized window attention, occupancy-pruned 2x upsampling | [decoder.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/decoder.py) | | Sparcubes vertex averaging, FlexiCubes, hole filling, cleanup | [mesh.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/hi3d/mesh.py) (NumPy) | Each pass refines the voxels of the previous pass's mesh, so the 1024 output runs the 512 transformer on a 64³ grid and then the 1024 transformer on a 128³ grid. The transformer runs in float16 with float32 norms, RoPE and modulation, as released; the decoder runs in float32 with float16 attention inputs and float32 softmax, matching the reference's flash attention. Submanifold convolutions gather each voxel's 27 neighbors into one im2col row; grid keys are int64 so the 1536³ decode does not collide. On an M4 Pro with 64 GB, a 512 refinement of the example dragon (12k voxels) takes about 3.5 minutes at 17 GB peak memory, and a 1024 refinement about 30 minutes at 30 GB, most of it in the second pass's full attention over 60k voxels. Latent noise comes from MLX's generator, so a seed does not reproduce a PyTorch run sample for sample. ## Validation Components were compared with the original PyTorch code run on CPU, with the unavailable spconv and flash-attn kernels replaced by reference implementations: | Component | Result | | --- | --- | | Image conditioning (float32) | max abs error 5.7e-6 | | Flow schedule | max abs error 6e-8 | | Transformer velocity, float32 | relative L2 error 2.2e-6 | | Transformer velocity, float16 | relative L2 error 1.3e-3 against float32 (the reference's own float16 run: 1.4e-3) | | `transformer_1024` on a 156³ grid (stretched RoPE), float32 / float16 | relative L2 error 2.6e-6 / 1.3e-3 | | Decoder occupancy, 1.44M voxels | 14 voxels differ at near-zero logits | | FlexiCubes and hole filling on reference decoder output | identical faces, vertices within 3e-6 | Run the component suite from the repository root: ```sh python -m pytest mlx_vlm/tests/test_extraction_models.py -q -k TestHi3D ``` ## Licenses This is an unofficial port. Hi3D code and weights are licensed under the [Hi3D Open License Agreement 1.0](https://huggingface.co/hi3d-official/Hi3Dv1.0/blob/main/LICENSE); products that integrate them commercially must display "Powered by Hi3D 1.0" and larger deployments need a separate license. Portions derived from Microsoft TRELLIS and Point Transformer V3 are MIT licensed, FlexiCubes is Apache-2.0, DINOv2 is Apache-2.0 and CLIP is MIT. The bundle carries `LICENSE`, `NOTICE`, `LICENSE-APACHE`, `LICENSE-CLIP` and `THIRD_PARTY_LICENSES.md`; keep them with redistributed weights. ## Bundle contents | Prefix | Source | Tensors | Dtypes | | --- | --- | ---: | --- | | `transformer_512` | hi3d-official/Hi3Dv1.0 | 512 | float16 (374), float32 (138) | | `transformer_1024` | hi3d-official/Hi3Dv1.0 | 512 | float16 (374), float32 (138) | | `vae` | hi3d-official/Hi3Dv1.0 | 141 | float32 (141) | | `conditioner.dino` | facebook/dinov2-large | 343 | float32 (343) | | `conditioner.clip` | openai/clip-vit-large-patch14 | 391 | float32 (391) | `conversion.json` lists every source file with its SHA-256 and each transform. The Hi3D tensors are unchanged in value and dtype. ## Local performance Hardware: Apple M4 Pro, 64 GiB unified memory. Input: Hi3D assets/example.png (436x640 RGBA), coarse mesh from mlx-community/sam-3d-objects-bf16 (seed 0), 1024 output, 25 Heun steps per pass, seed 0. | Stage | Seconds | | --- | ---: | | coarse_mesh | 77.6 | | load | 1.5 | | conditioning | 0.6 | | pass1_sampling | 189.1 | | pass1_decoding | 8.9 | | pass2_sampling | 1517.6 | | pass2_decoding | 45.0 | | postprocess | 8.7 | Peak memory: 30.1 GB. Output: 495,425 vertices, 1,000,000 faces.