Using LLMs
LLMs on FlyMy.AI are served through the same inference gateway as every other model, with one important difference: they only work over the streaming endpoint. The response arrives as a Server-Sent Events stream, chunk by chunk, and the final event carries the token counts you are billed on.
Every LLM endpoint is also exposed through an OpenAI-compatible route that works with the official OpenAI SDK, including non-streaming calls - see OpenAI-Compatible API. This page covers the native FlyMy.AI format.
Available models
| Model | Endpoint id | Notes |
|---|---|---|
| Gemini 2.5 Flash | flymyai/google-gemini-25-flash | Balanced speed and quality |
| Gemini 3 Flash (preview) | flymyai/google-gemini-3-flash-preview | Newer Flash generation |
| Gemini 3.1 Flash Lite (preview) | flymyai/google-gemini-31-flash-lite-preview | Cheapest and fastest |
| Gemini 3.1 Pro (preview) | flymyai/google-gemini-31-pro-preview | Highest quality, reasons before answering |
The model catalog displays names like google-gemini-2.5-flash, but the
endpoint id used by the API has no dots: google-gemini-25-flash. Copying
the displayed name into an API call returns 404 Project not found.
Quickstart
curl
curl -N -X POST \
https://api.flymy.ai/api/v1/flymyai/google-gemini-25-flash/predict/stream/ \
-H "X-API-KEY: fly-***" \
-F "prompt=Explain what a vector database is, briefly."
-N disables curl buffering so you see output as it arrives.
Python SDK
from flymyai import client
fma_client = client(apikey="fly-***")
stream = fma_client.stream(
model="flymyai/google-gemini-25-flash",
payload={"prompt": "Explain what a vector database is, briefly."},
)
for partial in stream:
chunk = partial.output_data.get("output")
if chunk:
print("".join(chunk), end="", flush=True)
print()
print("input tokens:", stream.stream_details.input_tokens)
print("output tokens:", stream.stream_details.output_tokens)
Raw SSE, any language
import json
import requests
url = "https://api.flymy.ai/api/v1/flymyai/google-gemini-25-flash/predict/stream/"
resp = requests.post(
url,
headers={"X-API-KEY": "fly-***"},
files={"prompt": (None, "Explain what a vector database is, briefly.")},
stream=True,
)
for line in resp.iter_lines(decode_unicode=True):
if not line or not line.startswith("data:"):
continue
event = json.loads(line[5:])
chunk = (event.get("output_data") or {}).get("output")
if chunk:
print("".join(chunk), end="", flush=True)
if event.get("stream_details"):
print("\nusage:", event["stream_details"])
Request format
| Field | Value |
|---|---|
| Method | POST |
| URL | https://api.flymy.ai/api/v1/flymyai/{endpoint_id}/predict/stream/ |
| Auth | X-API-KEY: fly-*** header |
| Body | multipart/form-data (or application/x-www-form-urlencoded) |
| Parameters | prompt (string, required), messages (JSON string, optional) |
Sending a JSON body returns 422 Field required even when the JSON is valid.
Send prompt and messages as form fields.
There is no temperature or max_tokens field - see
Limitations.
Chat history and images
The optional messages field carries a full conversation - system prompt,
prior turns, and image inputs - as a JSON-encoded string in the standard
Chat Completions format. When messages is present the model uses it and
ignores prompt; keep sending prompt anyway (the last user message, or an
empty string) because the field is required by the endpoint schema.
import json
from flymyai import client
fma_client = client(apikey="fly-***")
messages = [
{"role": "system", "content": "Answer in one sentence."},
{
"role": "user",
"content": [
{"type": "text", "text": "What food is in this image?"},
{
"type": "image_url",
"image_url": {"url": "https://example.com/photo.jpg"},
},
],
},
]
stream = fma_client.stream(
model="flymyai/google-gemini-25-flash",
payload={
"prompt": "What food is in this image?",
"messages": json.dumps(messages),
},
)
for partial in stream:
chunk = partial.output_data.get("output")
if chunk:
print("".join(chunk), end="", flush=True)
Images are content parts of type image_url, several per message if needed.
Both public HTTP(S) URLs and base64 data: URLs are accepted and passed to the
provider as-is; image input is billed as input tokens. Base64 data: URLs are
the most reliable way to send an image; remote URLs are fetched by the provider
and add its fetch latency. Base64 inflates the request body by about a third.
Response format
The response is an SSE stream with three kinds of events.
An acknowledgement, sent immediately, carrying the prediction id used to cancel the request:
event: {"prediction_id":"f9e443caf50e..."}
Zero or more content events. output is a list of strings; concatenate them and
append to what you have already received:
data: {"output_data":{"output":["A vector database stores "]},"status":200}
data: {"output_data":{"output":["embeddings and retrieves "]},"status":200}
A final event with empty output_data and the token usage for the request:
data: {"output_data":{},"status":200,"stream_details":{"input_tokens":15,"output_tokens":49}}
Use stream_details to reconcile your own usage tracking with your invoice -
these are the exact counts the charge is computed from.
Token usage and billing
You are billed per token, priced per 1M input tokens and per 1M output tokens. Charges are applied asynchronously: the request is recorded immediately, the ledger entry follows within minutes, and the wallet balance is recalculated on a periodic sync. A balance that has not moved seconds after a call is expected - check again after a few minutes rather than assuming the call was free.
Per-request usage is available from the model usage endpoint.
Limitations
Read these before designing an integration - they are the constraints most likely to affect your architecture.
No server-side conversation state. The API stores no history between
requests. To hold a conversation, keep the transcript yourself and resend it in
the messages field on every turn - see
Chat history and images.
No prompt caching. Every request is billed for its full input, including any prefix you have sent before. In a chat, where the transcript is resent each turn, input tokens dominate: expect roughly three input tokens for every output token. Keep transcripts bounded if cost matters.
No sampling controls. Temperature, top-p, stop sequences, and output length
limits are not exposed. Steer the model through the prompt instead, for example
Answer in under 120 words.
Streaming only. The non-streaming /predict route is not available for LLM
projects. It answers with HTTP 200 and an error inside the body:
{"detail":"Predict is not available for current pipeline. Use /predict/stream route instead!","status_code":421}
Check status_code inside the payload, not just the HTTP status. If you need a
plain request/response call, use the
OpenAI-compatible endpoint - its sync mode collects the
stream server-side and returns one complete response.
Errors
| What you see | Meaning |
|---|---|
403 API key does not have access to the project | The model is private, or the endpoint id is wrong |
403 Insufficient funds | Wallet balance is negative - top up |
404 Project not found | Endpoint id does not exist, often dots copied from the display name |
422 Field required | prompt was sent as JSON instead of form data |
HTTP 200 with status_code: 421 | You called /predict instead of /predict/stream/ |
Latency
Measured on production against Gemini 3.1 Flash Lite, three concurrent chat sessions over 30 minutes, 1062 requests, no failures:
| Metric | p50 | p95 | p99 |
|---|---|---|---|
| Acknowledgement | 0.41s | 1.06s | 2.15s |
| First content chunk | 0.91s | 1.64s | 3.53s |
| Complete answer | 1.49s | 2.23s | 3.53s |
Reasoning-heavy prompts on Pro models delay the first chunk by several seconds while the model thinks - the stream stays open and content follows normally.
API reference
The gateway publishes an OpenAPI schema, and every model publishes its own:
- Interactive: api.flymy.ai/docs
- Per-model schema:
https://api.flymy.ai/api/v1/flymyai/{endpoint_id}/openapi.json