feat(ai): add experimental generate3DModel() with a fal adapter and a game asset tutorial #2186
main fromt3/image-to-3d-assets·merged by AlemTuzlak View on GitHubYou can now turn text, an image, or an existing mesh into a 3D model file (GLB and more) with generate3DModel(). fal3DModel() runs the fal 3D endpoints, and hooks stream the job to the UI in 7 frameworks. A new example and tutorial show a full game asset pipeline: one line of text gives a low-poly GLB with PBR textures, and a second line paints the same mesh with a new skin.
🎯 Changes#
Core (@tanstack/ai), experimental
generate3DModel()uses a job API likegenerateVideo. It has 3 modes: submit only (returns{ jobId }),stream: true(SSE for the UI), andwait: true(returns the finished model, for server pipelines).get3DModelJobStatus()reads a job by id.- A media prompt can carry a new
{ type: 'model3d', source }part, so a step can take the mesh of an earlier step. The part is inMediaPromptPartonly, not in chatContentPart. - The result is
{ jobId, url, format, files?, previewImageUrl?, usage? }.urlis the best GLB.filesholds every mesh and texture file with arole.
fal (@tanstack/ai-fal)
fal3DModel('<endpoint>')knows 30 fal 3D endpoints, including Meshy 7.1, Meshy 7 Retexture, and Tripo Remesh.modelOptionsis typed for each endpoint, and the prompt parts are narrowed for each endpoint. Any other fal id also works, with no types.- Prompt parts go to the input field of each endpoint (from the generated image and mesh field maps). Each known endpoint lists its image fields, checked against the SDK types. A part that lands on another field, or that the endpoint cannot use, throws before the job starts.
- One output mapper covers every endpoint.
@fal-ai/clientgoes from 1.10.1 to 1.11.0. 1.11.0 has no types for 41 image and video endpoints that fal still serves (for example Veo 3.1 first-last-frame and Kling 1.x image-to-video). A hand-kept table insrc/image/legacy-image-field-overrides.tskeeps their image input fields.
Hooks: useGenerate3DModel (React, Solid, Vue, Octane), createGenerate3DModel (Svelte, Remix), injectGenerate3DModel (Angular).
Examples and docs
- New
examples/react/game-asset-pipelineand the tutorialdocs/tutorials/game-asset-pipeline.md. The steps are flux-2 concept art (or an image that you upload), a Meshy Smart Topology mesh (meshy/t2.5/image-to-3d) with about 10,000 triangles, a flux-2 edit skin, and a Meshy 7 retexture. An uploaded image goes to fal as adata:URL, so it needs no upload route. - A "3D" tab in
ts-react-mediawith a three.js viewer and a Retexture tool. - New pages
docs/media/3d-model-generation.mdand3d-model-generation-advanced.md. Updateddocs/adapters/fal.md,generation-hooks.md, Overview, Quick Start, andconfig.json. - Changesets: one for the fal client update (ai-fal), and minor for ai, ai-fal, ai-client, ai-react, ai-solid, ai-vue, ai-svelte, ai-angular, ai-octane, ai-remix.
Other:
- The otel middleware now records the 3D model URL on media spans, the same as video. The forge fixer agent pushed this commit.
scripts/generate-fal-image-field-map.tsnow also writes the mesh field map. It also loads@typescript/typescript6, because it could not run on TS 7.
✅ Checklist#
- I have followed the steps in the Contributing guide.
- I have tested code changes locally with
pnpm run test:pr, or these tests do not apply to this pull request. (Not run. Light checks only, see Testing. CI runs the full set.) - I fully understand the code in this pull request, including any code generated with AI assistance.
- Docs: I updated
docs/for this change, or this change is not user-facing. - Changeset: I added a changeset (
pnpm changeset), or this PR does not change a published package.
🚀 Release Impact#
- This change affects published code, and I have generated a changeset.
- This change is docs/CI/dev-only (no release).
Testing#
Commands run (all pass)
- Unit tests, changed files only:
packages/ai:generate-3d-model,media-prompt,modality-types,bound-activities(64 tests), andmiddlewares/otel(62 tests).packages/ai-fal:model3d-adapter,mesh-field-map,image-inputs,video-adapter(90 tests).- The generation test file of each hook package (React, Solid, Vue, Svelte, Angular).
- After the last review round:
generate-3d-model(15 tests), the 4 ai-fal files above, the 5 hook test files, and the typecheck of all 7 hook packages and both examples. tscfor the changed packages, all provider adapters that read prompts, andexamples/react/game-asset-pipeline.testing/e2eandts-react-media: no type errors in the 3D files. The only errors are missing builds of other packages.oxlintandoxfmton the changed files, andpnpm test:docs.pnpm install --frozen-lockfilewith the new lockfile entry.
Review fixes in 8d7e207b2: all 43 review threads have a reply and are resolved. This push:
- Rejects fal image parts that land on a field the endpoint does not have.
- Creates no job for an aborted run, and cancels the job at abort while a stream consumer is paused.
- Clears the mesh and the skin in the game asset example when the concept changes. The upload there takes PNG and JPEG only, because Meshy takes only these formats.
- Keeps the last model after a failed retexture in the 3D media tab. Stop there now frees the run lock.
- Adds
generate3DModelto the otel docs table.
The hooks now also pass persistence: false at runtime, and file read errors show in both examples.
Not run locally (CI runs them): the E2E spec testing/e2e/tests/3d-model-generation.spec.ts, pnpm test:pr, knip, sherif, full builds, and coverage.
Real run with fal, in the browser
The first full run used Meshy v6 for the mesh and TRELLIS-2 for the retexture. I ran all 4 steps of the example against fal:
- Concept art: flux-2 drew a wooden treasure chest.
- Mesh: Meshy v6 returned a GLB with 19,769 triangles and PBR textures (base color, metallic-roughness, normal), plus a preview image.
- Skin: flux-2 edit drew the same chest in gold with red gems.
- Retexture: TRELLIS-2 returned a GLB with the same 19,769 triangles and new textures.
A second run used an uploaded image: the front view of an armored orc warrior. Meshy v6 accepted the data: URL and returned a 25,475-triangle GLB with PBR textures. flux-2 edit drew a gold skin from the same upload, and TRELLIS-2 painted the same mesh with it.
The same run found that fal no longer serves 7 3D endpoints that @fal-ai/client 1.10.1 still types: Hunyuan3D Smart Topology and Part, Meshy v5 Retexture, UltraShape, TripoSR, Hunyuan3D 2.1, and Hunyuan3D 3 Sketch. fal answers 404 for them. They are not in Fal3DKnownModel, the docs, or the examples.
After the move to Smart Topology, a direct fal run of meshy/t2.5/image-to-3d on the orc image gave a GLB with 10,312 triangles and PBR textures in 297 s. I did not run the app again with Smart Topology and Meshy 7 Retexture.
Manual test
- Run
pnpm install, thenpnpm --filter game-asset-pipeline dev. - Open http://localhost:3100 and paste a fal key.
- Click Draw, or upload an image of one object on a plain background.
- Click Make mesh, then Draw skin, then Retexture.
- Download the GLB from step 2 and from step 4. Open both in Blender or https://gltf-viewer.donmccurdy.com.
How this PR makes testing easy
- Unit tests for the activity (all 3 modes, abort before submit, abort while a stream consumer is paused, timeout, failed jobs), the fal input routing and output mapping, and each hook.
- An E2E spec with an in-route fal queue mock.
- The
game-asset-pipelineexample and thets-react-media3D tab.
Risk / rollback#
The risk is low for current users. All new APIs are experimental and additive. The @fal-ai/client 1.11.0 update removes the typed modelOptions for Veo, Kling 1.6, and Whisper, because the SDK has no types for them. These models still run. The main shared types that changed are MediaPromptPart and MediaPromptModality, which now include model3d. A provider adapter that switches over every part type has a new case. The typecheck of all adapters passes. To roll back, revert the PR.
Public API change#
Before: no 3D activity.
After
import { generate3DModel } from '@tanstack/ai'
import { fal3DModel } from '@tanstack/ai-fal'
const model = await generate3DModel({
adapter: fal3DModel('meshy/v7.1/image-to-3d'),
prompt: [{ type: 'image', source: { type: 'url', value: conceptUrl } }],
modelOptions: { topology: 'quad', target_polycount: 10_000 },
threadId: 'chest',
wait: true,
})
console.log(model.url) // the GLB
const { generate, result } = useGenerate3DModel({
connection: fetchServerSentEvents('/api/model3d'),
threadId: 'chest',
})
Summary by CodeRabbit#
- New Features
- Added experimental 3D model generation with fal.ai, including text, image, and model inputs; job progress, status checks, results, and cancellation.
- Added generation helpers for Angular, React, Solid, Svelte, Vue, and Remix.
- Added examples for generating, previewing, and downloading 3D models, including a game-asset pipeline for concept art, meshes, and retextured models.
- Documentation
- Added 3D generation guides, a game-asset tutorial, and fal.ai adapter documentation covering generation options, workflows, and model files.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
AlemTuzlak pushed8d7e207
Not changed. CONTRIBUTING.md tells contributors to run pnpm run build:all once, and the neighbor examples (for example generate-image) have the same two steps with no build step.
Fixed in 8d7e207b2. A new concept now clears the mesh and the skin, so Retexture needs a mesh and a skin from the same concept.
Fixed in af18540ae: the run lock is set before the file read.
Fixed in 8d7e207b2. After a failed retexture, the last model, its downloads, and Retexture stay on screen, with the error above them.
Fixed. 8d7e207b2 replaces the count with a check of the mapped fields against the image fields of each endpoint.
Fixed in af18540ae: 3D hooks no longer take persistence.
Not changed here. InferGenerationOutputFromReturn comes from ai-client on main, and every generation hook has the same behavior. This needs one change for all hooks, in its own PR.
Fixed in af18540ae: 3D hooks no longer take persistence.
Fixed in af18540ae: 3D hooks no longer take persistence.
Fixed in af18540ae: 3D hooks no longer take persistence.
Fixed in af18540ae: 3D hooks no longer take persistence.
Fixed in af18540ae: createJob keeps watching the submission and cancels a late job id.
Fixed in 8d7e207b2. createJob now rejects before it calls the adapter when the signal is already aborted. A test checks that no job is created.
Fixed in 8d7e207b2. An abort listener now cancels the job at once, also while a stream consumer is paused at a yield. A test covers it.
Not changed here. generateImage and generateVideo also do not race applyGenerationResultTransforms against the deadline. A change belongs in all activities at once, in its own PR.
Fixed in 8d7e207b2. The tutorial is rebuilt from the app code, so it has the same change: a new concept clears the mesh and the skin.
Not changed. CONTRIBUTING.md tells contributors to run pnpm run build:all once, and the neighbor examples (for example generate-image) have the same two steps with no build step.
Fixed in bcaa66b31: callWithByok returns a successful SSE response unchanged.
AlemTuzlakreplied on
packages/ai/src/activities/generate3DModel/index.tsline 307 Only here. This is not on GitHub.Fixed in af18540ae: createJob keeps watching the submission and cancels a late job id.
AlemTuzlakreplied on
packages/ai-fal/src/model3d/model3d-inputs.tsline 112 Only here. This is not on GitHub.Fixed. 8d7e207b2 replaces the count with a check of the mapped fields against the image fields of each endpoint.
Sign in to join this conversation.
packages/ai/src/middlewares/otel.tsline 79 Only here. This is not on GitHub.Fixed. The code part landed earlier (otel.ts reads the 3D url). 8d7e207b2 adds generate3DModel to the otel docs table.