Microsoft Graph API won't work in production - php

Question

I have my site set up so I can use Oauth for single sign on and added access to be able to post calendar entries made on my site to the user's outlook calendar.
Everything works on my dev environment but when I go to my calendar page in production it throws this error:
https://graph.microsoft.com/v1.0/me/calendarsresulted in a401 Unauthorized` response: { "error": { "code": "InvalidAuthenticationToken", "message": "Access token validation failure."
I am using the same token I get from oAuth for the calendars. Is that not the right way to do it? And if not how do I go about the right way to do it and why would it work?
here is a part of my oauth page
// Get an access token using the authorization code grant
$accessToken = $provider->getAccessToken('authorization_code', [
'code' => $_GET['code']
]);
$_SESSION['refresh_token'] =$accessToken->getRefreshToken();
$graph = new Graph();
$graph->setAccessToken($accessToken);
//the sesion token for calendars expires faster than our token to stay in connect, so if it expires have the user sign back in
try {
$user = $graph->createRequest("GET", "/me/calendars")
->execute();
} catch (Exception $e) { //FAILS HERE
// sessionTimeout(); exit;
var_dump($e);
}
EDIT
throwing the token i get back into JWT I see that when in development my token has all the information including about what scopes i have. But in production the token is missing the "scp" field. I have no idea what I am doing wrong.
EDIT 2
I thought maybe I would need a refresh token but that still gives me the same error.