# Using LLMs

> Run Gemini and other LLMs on FlyMy.AI: streaming over SSE, chat history and image inputs, request format, token usage and billing, limitations.

LLMs on FlyMy.AI are served through the same inference gateway as every other
model, with one important difference: **they only work over the streaming
endpoint**. The response arrives as a Server-Sent Events stream, chunk by chunk,
and the final event carries the token counts you are billed on.

Every LLM endpoint is also exposed through an OpenAI-compatible route that
works with the official OpenAI SDK, including non-streaming calls - see
[OpenAI-Compatible API](https://docs.flymy.ai/guides/llm-openai-compat.md). This page covers the native
FlyMy.AI format.

Grok 4.7, DeepSeek V4.1 Flash and Kimi K3.1 Pro are the exceptions: they
answer on `/predict` and the OpenAI-compatible route, not on
`/predict/stream/`, and call function tools - see [Grok 4.7](https://docs.flymy.ai/guides/grok-4-7.md),
[DeepSeek V4.1 Flash](https://docs.flymy.ai/guides/deepseek-v4-1-flash.md) and [Kimi K3.1 Pro](https://docs.flymy.ai/guides/kimi-k3-1-pro.md).
Grok 4.7 and DeepSeek V4.1 Flash also take sampling controls; Kimi K3.1 Pro
sets its own.

Claude, GPT and Gemini (such as `flymyai/claude-haiku-5-5`) answer on
`/predict`, the OpenAI-compatible route and their maker's own API with the
official SDKs - see [Claude, GPT and Gemini](https://docs.flymy.ai/guides/claude-gpt-gemini.md).

## Available models

| Model | Endpoint id | Notes |
| --- | --- | --- |
| Gemini 2.5 Flash | `flymyai/google-gemini-25-flash` | Balanced speed and quality |
| Gemini 3 Flash (preview) | `flymyai/google-gemini-3-flash-preview` | Newer Flash generation |
| Gemini 3.1 Flash Lite (preview) | `flymyai/google-gemini-31-flash-lite-preview` | Cheapest and fastest |
| Gemini 3.1 Pro (preview) | `flymyai/google-gemini-31-pro-preview` | Highest quality, reasons before answering |
| Grok 4.7 | `flymyai/grok-4_7` | Reasoning effort from low to xhigh, private mode; see [Grok 4.7](https://docs.flymy.ai/guides/grok-4-7.md) |
| DeepSeek V4.1 Flash | `flymyai/deepseek-v4_1-flash` | Fast, low-cost reasoning with function tools, a priority tier and private mode; see [DeepSeek V4.1 Flash](https://docs.flymy.ai/guides/deepseek-v4-1-flash.md) |
| Kimi K3.1 Pro | `flymyai/kimi-k3_1-pro` | Moonshot's thinking model for agents, coding and long documents, with function tools, a priority tier and private mode; see [Kimi K3.1 Pro](https://docs.flymy.ai/guides/kimi-k3-1-pro.md) |

> **Caution: Endpoint ids contain no dots**
>
> The model catalog displays names like `google-gemini-2.5-flash`, but the
> endpoint id used by the API has **no dots**: `google-gemini-25-flash`. Copying
> the displayed name into an API call returns `404 Project not found`.

## Quickstart

### curl

```bash
curl -N -X POST \
  https://api.flymy.ai/api/v1/flymyai/google-gemini-25-flash/predict/stream/ \
  -H "X-API-KEY: fly-***" \
  -F "prompt=Explain what a vector database is, briefly."
```

`-N` disables curl buffering so you see output as it arrives.

### Python SDK

```python
from flymyai import client

fma_client = client(apikey="fly-***")

stream = fma_client.stream(
    model="flymyai/google-gemini-25-flash",
    payload={"prompt": "Explain what a vector database is, briefly."},
)

for partial in stream:
    chunk = partial.output_data.get("output")
    if chunk:
        print("".join(chunk), end="", flush=True)

print()
print("input tokens:", stream.stream_details.input_tokens)
print("output tokens:", stream.stream_details.output_tokens)
```

### Raw SSE, any language

```python
import json
import requests

url = "https://api.flymy.ai/api/v1/flymyai/google-gemini-25-flash/predict/stream/"
resp = requests.post(
    url,
    headers={"X-API-KEY": "fly-***"},
    files={"prompt": (None, "Explain what a vector database is, briefly.")},
    stream=True,
)

for line in resp.iter_lines(decode_unicode=True):
    if not line or not line.startswith("data:"):
        continue
    event = json.loads(line[5:])
    chunk = (event.get("output_data") or {}).get("output")
    if chunk:
        print("".join(chunk), end="", flush=True)
    if event.get("stream_details"):
        print("\nusage:", event["stream_details"])
```

## Request format

| Field | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://api.flymy.ai/api/v1/flymyai/{endpoint_id}/predict/stream/` |
| Auth | `X-API-KEY: fly-***` header |
| Body | `multipart/form-data` (or `application/x-www-form-urlencoded`) |
| Parameters | `prompt` (string, required), `messages` (JSON string, optional) |

> **Caution: The body is form data, not JSON**
>
> Sending a JSON body returns `422 Field required` even when the JSON is valid.
> Send `prompt` and `messages` as form fields.

There is no `temperature` or `max_tokens` field - see
[Limitations](#limitations).

## Chat history and images

The optional `messages` field carries a full conversation - system prompt,
prior turns, and image inputs - as a **JSON-encoded string** in the standard
Chat Completions format. When `messages` is present the model uses it and
ignores `prompt`; keep sending `prompt` anyway (the last user message, or an
empty string) because the field is required by the endpoint schema.

```python
import json
from flymyai import client

fma_client = client(apikey="fly-***")

messages = [
    {"role": "system", "content": "Answer in one sentence."},
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What food is in this image?"},
            {
                "type": "image_url",
                "image_url": {"url": "https://example.com/photo.jpg"},
            },
        ],
    },
]

stream = fma_client.stream(
    model="flymyai/google-gemini-25-flash",
    payload={
        "prompt": "What food is in this image?",
        "messages": json.dumps(messages),
    },
)

for partial in stream:
    chunk = partial.output_data.get("output")
    if chunk:
        print("".join(chunk), end="", flush=True)
```

Images are content parts of type `image_url`, several per message if needed.
Both public HTTP(S) URLs and base64 `data:` URLs are accepted and passed to the
provider as-is; image input is billed as input tokens. Base64 `data:` URLs are
the most reliable way to send an image; remote URLs are fetched by the provider
and add its fetch latency. Base64 inflates the request body by about a third.

## Response format

The response is an SSE stream with three kinds of events.

An acknowledgement, sent immediately, carrying the prediction id used to cancel
the request:

```
event: {"prediction_id":"f9e443caf50e..."}
```

Zero or more content events. `output` is a list of strings; concatenate them and
append to what you have already received:

```
data: {"output_data":{"output":["A vector database stores "]},"status":200}
data: {"output_data":{"output":["embeddings and retrieves "]},"status":200}
```

A final event with empty `output_data` and the token usage for the request:

```
data: {"output_data":{},"status":200,"stream_details":{"input_tokens":15,"output_tokens":49}}
```

Use `stream_details` to reconcile your own usage tracking with your invoice -
these are the exact counts the charge is computed from.

## Token usage and billing

You are billed per token, priced per 1M input tokens and per 1M output tokens.
Charges are applied asynchronously: the request is recorded immediately, the
ledger entry follows within minutes, and the wallet balance is recalculated on a
periodic sync. A balance that has not moved seconds after a call is expected -
check again after a few minutes rather than assuming the call was free.

Per-request usage is available from the [model usage endpoint](https://docs.flymy.ai/api/usage.md).

## Limitations

Read these before designing an integration - they are the constraints most
likely to affect your architecture.

**No server-side conversation state.** The API stores no history between
requests. To hold a conversation, keep the transcript yourself and resend it in
the `messages` field on every turn - see
[Chat history and images](#chat-history-and-images).

**No prompt caching.** Every request is billed for its full input, including any
prefix you have sent before. In a chat, where the transcript is resent each
turn, input tokens dominate: expect roughly three input tokens for every output
token. Keep transcripts bounded if cost matters.

**No sampling controls.** Temperature, top-p, stop sequences, and output length
limits are not exposed. Steer the model through the prompt instead, for example
`Answer in under 120 words.`

**Streaming only.** The non-streaming `/predict` route is not available for LLM
projects. It answers with HTTP 200 and an error inside the body:

```json
{"detail":"Predict is not available for current pipeline. Use /predict/stream route instead!","status_code":421}
```

Check `status_code` inside the payload, not just the HTTP status. If you need a
plain request/response call, use the
[OpenAI-compatible endpoint](https://docs.flymy.ai/guides/llm-openai-compat.md) - its sync mode collects the
stream server-side and returns one complete response.

## Errors

| What you see | Meaning |
| --- | --- |
| `403 API key does not have access to the project` | The model is private, or the endpoint id is wrong |
| `403 Insufficient funds` | Wallet balance is negative - top up |
| `404 Project not found` | Endpoint id does not exist, often dots copied from the display name |
| `422 Field required` | `prompt` was sent as JSON instead of form data |
| HTTP 200 with `status_code: 421` | You called `/predict` instead of `/predict/stream/` |

## Latency

Measured on production against Gemini 3.1 Flash Lite, three concurrent chat
sessions over 30 minutes, 1062 requests, no failures:

| Metric | p50 | p95 | p99 |
| --- | --- | --- | --- |
| Acknowledgement | 0.41s | 1.06s | 2.15s |
| First content chunk | 0.91s | 1.64s | 3.53s |
| Complete answer | 1.49s | 2.23s | 3.53s |

Reasoning-heavy prompts on Pro models delay the first chunk by several seconds
while the model thinks - the stream stays open and content follows normally.

## API reference

The gateway publishes an OpenAPI schema, and every model publishes its own:

- Interactive: [api.flymy.ai/docs](https://api.flymy.ai/docs)
- Per-model schema: `https://api.flymy.ai/api/v1/flymyai/{endpoint_id}/openapi.json`

## Related guides

- [OpenAI-Compatible API](https://docs.flymy.ai/guides/llm-openai-compat.md)
- [Image Generation](https://docs.flymy.ai/guides/generate-images.md)
- [Video Generation](https://docs.flymy.ai/guides/generate-videos.md)
- [API Overview](https://docs.flymy.ai/api/overview.md)

## Typed decisions

For fast classification, routing, triage, fixed-enum extraction and yes/no gates, use [TypeSafe Jev 1.13](https://docs.flymy.ai/guides/typesafe-jev). It accepts `state` and typed `questions` through its JSON Decisions endpoint and returns Choice, Noul probability or Score answers. It is not a chat model and does not generate prose, code or media. Both MCP servers expose `make_typed_decision`; agents discover `typesafe_jev.decide`.
