We updated a custom schema to include an additional optional field but did not change the version number.
We then updated the schema of the events table in Big Query to include the additional field.
However, for some reason in prod the field does not get populated by the bq loader in Big Query even though it is passed through enrichment fine. The bizarre thing is that with the same steps above, it is working in our test env and the bq loader is loading the field correctly in BQ.
For any modification (e.g., addition) to a schema you should increase the schema version - this ensures that the loaders can deal with it appropriately and automatically mutate the table for the new version.
In terms of why it isn’t being loaded - that’s hard to say, particularly if you’ve manually modified the table rather than having the mutator do it - but looking at the failed inserts pubsub topic (if the event is passing enrichment) should give you some indication as to why it can’t be inserted into BigQuery.