Apparently jackson + friends make assumptions when it comes to parsing raw -> ZonedDateTime. It appears that the ZonedDateTime for createInstant when deserializing timestamps are not being handled properly somehow, so you need custom code. We have our own Jackson serializer and deserializer for ZonedDateTime to ensure we always get milli in and milli out.

Here's the code (Apache2 license) and the maven repository links if it's helpful.

Thanks for the reply @dan.
We ended up storing sub ID on our side. We thought that might gives us more flexibility down the road (not sure about this but if it doesn't work then we'll have an extra column in the db; a small price to pay IMHO).

"Where to map the ID?" was a bit headache for us as we didn't want to miss it or make an extra call by polling ourselves. Since we are using .NET Core, we ended up hooking into OnUserInformationReceived event, one of many from OpenIdConnectEvents. Hopefully, I'm not doing something really stupid.