Which Fields in an API Response Are Actually Optional?
You're integrating an API. The docs are thin or out of date, so you do the sensible thing: hit the endpoint, look at the response, and write your types from what you see. Every field is there, so every field goes in as required. The integration works, the tests pass, and three weeks later it throws on a record where email simply isn't present.
The mistake wasn't carelessness. It's that the question "is this field optional?" has no answer inside a single record. One sample tells you a field was present once. Optionality is a property of the whole set, and you can only see it by looking at many records at the same time.
Why one sample systematically misleads
Look at a single user object and you'll see a name, an email, an avatar URL, a bio. Nothing marks any of them as conditional, because nothing can — a JSON object is a snapshot, not a contract. The fields that happened to be populated for that record look exactly like the fields that are always populated.
What makes this worse than an ordinary unknown is that the sample you grab is rarely a random one. It's usually the first record, or your own test account, or the demo data the API provider seeded — and those are the most complete records in the system. Test accounts get filled in. Real users skip the bio, never upload an avatar, sign up through a flow that doesn't collect a phone number. So the one record you inspected is close to a best case, and every assumption you derive from it is optimistic in the same direction.
What actually varies between records
Optional fields are the most common case, but they're not the only thing that only shows up in aggregate.
Type drift. A field arrives as a number in most records and a string in a few. This usually means two code paths produce the response — an older serializer that stringified IDs and a newer one that doesn't, or a value that passes through a template in one branch. Typed clients break on it hard, and it's invisible unless you compare the same field across records.
Always-null fields. A field is present in every record and null in every record. You cannot infer its type, and it's important to know that rather than guessing "string" and moving on — it usually means your sample doesn't cover whatever condition populates it.
Missing versus present-but-null. These are different states and often need different handling. A key that's absent and a key set to null reach your code as two different things in most languages, and a server that produces both for the same field is usually doing so unintentionally.
Two names for one field. Some records carry user_id and others carry userId. The tell is that no single record has both — they split cleanly across the set, which is the signature of two serialization paths rather than two genuine fields. Names that do coexist in a record, like title and subtitle, are just different fields.
Why the usual tools don't answer this
The instinct is to reach for a JSON diff, and it's the wrong shape. Diff tools take exactly two documents, a left and a right. That's built for "what changed between v1 and v2," not "what varies across forty records." Answering optionality with pairwise diffs means comparing every combination — 780 runs for forty records — and then holding the results in your head.
The workaround people land on is a shell loop that pulls N responses and diffs each against the first. It finds fields that record one happens to have and silently misses any field that only appears in records seven and twenty-three. It also anchors everything to whichever record you fetched first, which is exactly the biased sample the whole exercise was meant to escape.
Type generators look closer, since they do accept multiple samples. But their job is to emit something that compiles against every sample, so they resolve inconsistency rather than report it. A field missing from some records becomes email?: string. A field that's sometimes a string and sometimes a number becomes a union. That's correct for code generation and precisely backwards for auditing — the variation you were trying to find is the thing that got absorbed into the output. Worse, the ? looks like a documented fact when it's actually an artifact of your particular sample.
Getting a sample that can tell you the truth
Since the answer depends entirely on the records you look at, the sample matters more than the tooling.
Pull from a list endpoint rather than fetching single records, and page deep enough to leave the newest data — recently created records skew toward whatever the current signup flow collects. Include records you know are unusual: the oldest ones in the system, accounts created before a schema migration, entries from a bulk import. Deliberately include a sparse record if you can find one, since those carry the information a complete record can't.
And treat the result as evidence, not proof. "Present in 40 of 40" means you didn't observe it missing, not that it can't be. What it does give you is a specific, checkable list of questions for whoever maintains the API — "is avatar_url guaranteed, or did my sample just happen to have it?" is a much better question than "are there any optional fields?", and far more likely to get a real answer.
Doing it in one pass
FreeToolDev's Bulk JSON Field Consistency Checker takes the whole set at once — paste a list endpoint's array response as-is, or several records separated by a divider — and reports which fields appear in every record, which are only sometimes there, which change type, which are null throughout, and which pairs of names never co-occur. It runs entirely in your browser, which matters when the payload is real customer data.
Once you know what actually varies, encode it: write a schema that marks the genuinely optional fields optional and run future batches against it with a bulk schema validator, so the next shape change shows up as a failed check instead of a bug report. For why that drift happens in the first place, and why it stays invisible for so long, see how an API response changes shape without anyone noticing.