Cloud Composer throwing InvalidToken after adding another node
I recently added a few new DAGs to production airflow and as a result decided to scale up the number of nodes in the Composer pool. After doing so I got the error: Can't decrypt _val for key=<KEY>, invalid token or value. This happens now for every single DAG that uses variables. It's not the same key either, it depends on what variables the DAG needs.
I immediately scaled Composer back down to 3 nodes and the problem persisted.
I have tried re-saving all of the Variables, recreating them in the UI (which says they are all valid), recreating them in the CLI (which lists invalid for every single one).
I have also tried updating configuration to try and reboot the server, and manually stopping the VM instances.
Composer also seems to negate the ability to update the Fernet Key, so I can't try and use a new one. For some reason it appears that the permanent one Composer has assigned is now invalid.
Is there anything else that can be done to remedy that problem short of recreating the environment?
google-cloud-platform airflow google-cloud-composer
asked Nov 14 '18 at 0:10
KJSKJS
426
add a comment |
I recently added a few new DAGs to production airflow and as a result decided to scale up the number of nodes in the Composer pool. After doing so I got the error: Can't decrypt _val for key=<KEY>, invalid token or value. This happens now for every single DAG that uses variables. It's not the same key either, it depends on what variables the DAG needs.
I immediately scaled Composer back down to 3 nodes and the problem persisted.
I have tried re-saving all of the Variables, recreating them in the UI (which says they are all valid), recreating them in the CLI (which lists invalid for every single one).
I have also tried updating configuration to try and reboot the server, and manually stopping the VM instances.
Composer also seems to negate the ability to update the Fernet Key, so I can't try and use a new one. For some reason it appears that the permanent one Composer has assigned is now invalid.
Is there anything else that can be done to remedy that problem short of recreating the environment?
google-cloud-platform airflow google-cloud-composer
asked Nov 14 '18 at 0:10
KJSKJS
426
I've never used Composer but is there a way to view the scheduler logs separate from the webserver, and if so does the error happen in both?
– joeb
Nov 14 '18 at 2:11
@joeb Stackdriver has the logs split for the scheduler, it unfortunately throws the same error as it tries to load the DAGs. Nothing has changed this morning, I think my best course of action is to recreate the environment.
– KJS
Nov 14 '18 at 16:48
add a comment |
I recently added a few new DAGs to production airflow and as a result decided to scale up the number of nodes in the Composer pool. After doing so I got the error: Can't decrypt _val for key=<KEY>, invalid token or value. This happens now for every single DAG that uses variables. It's not the same key either, it depends on what variables the DAG needs.
I immediately scaled Composer back down to 3 nodes and the problem persisted.
I have tried re-saving all of the Variables, recreating them in the UI (which says they are all valid), recreating them in the CLI (which lists invalid for every single one).
I have also tried updating configuration to try and reboot the server, and manually stopping the VM instances.
Composer also seems to negate the ability to update the Fernet Key, so I can't try and use a new one. For some reason it appears that the permanent one Composer has assigned is now invalid.
Is there anything else that can be done to remedy that problem short of recreating the environment?
google-cloud-platform airflow google-cloud-composer
asked Nov 14 '18 at 0:10
KJSKJS
426
I recently added a few new DAGs to production airflow and as a result decided to scale up the number of nodes in the Composer pool. After doing so I got the error: Can't decrypt _val for key=<KEY>, invalid token or value. This happens now for every single DAG that uses variables. It's not the same key either, it depends on what variables the DAG needs.
I immediately scaled Composer back down to 3 nodes and the problem persisted.
I have tried re-saving all of the Variables, recreating them in the UI (which says they are all valid), recreating them in the CLI (which lists invalid for every single one).
I have also tried updating configuration to try and reboot the server, and manually stopping the VM instances.
Composer also seems to negate the ability to update the Fernet Key, so I can't try and use a new one. For some reason it appears that the permanent one Composer has assigned is now invalid.
Is there anything else that can be done to remedy that problem short of recreating the environment?
google-cloud-platform airflow google-cloud-composer
google-cloud-platform airflow google-cloud-composer
asked Nov 14 '18 at 0:10
KJSKJS
426
asked Nov 14 '18 at 0:10
KJSKJS
426
asked Nov 14 '18 at 0:10
KJSKJS
426
asked Nov 14 '18 at 0:10
KJSKJS
426
asked Nov 14 '18 at 0:10
KJSKJS
426
426
I've never used Composer but is there a way to view the scheduler logs separate from the webserver, and if so does the error happen in both?
– joeb
Nov 14 '18 at 2:11
@joeb Stackdriver has the logs split for the scheduler, it unfortunately throws the same error as it tries to load the DAGs. Nothing has changed this morning, I think my best course of action is to recreate the environment.
– KJS
Nov 14 '18 at 16:48
add a comment |
I've never used Composer but is there a way to view the scheduler logs separate from the webserver, and if so does the error happen in both?
– joeb
Nov 14 '18 at 2:11
@joeb Stackdriver has the logs split for the scheduler, it unfortunately throws the same error as it tries to load the DAGs. Nothing has changed this morning, I think my best course of action is to recreate the environment.
– KJS
Nov 14 '18 at 16:48
I've never used Composer but is there a way to view the scheduler logs separate from the webserver, and if so does the error happen in both?
– joeb
Nov 14 '18 at 2:11
I've never used Composer but is there a way to view the scheduler logs separate from the webserver, and if so does the error happen in both?
– joeb
Nov 14 '18 at 2:11
@joeb Stackdriver has the logs split for the scheduler, it unfortunately throws the same error as it tries to load the DAGs. Nothing has changed this morning, I think my best course of action is to recreate the environment.
– KJS
Nov 14 '18 at 16:48
@joeb Stackdriver has the logs split for the scheduler, it unfortunately throws the same error as it tries to load the DAGs. Nothing has changed this morning, I think my best course of action is to recreate the environment.
– KJS
Nov 14 '18 at 16:48
add a comment |
1 Answer
1
active
oldest
votes
I managed to fix this problem by adding a new python package. It seems that adding a package is the only way to really "reboot" the environment. The reboot invalidated all of my variables and connections when it had finished but I was able to just add those back in rather than having to recreate the entire environment.
Heard back about this issue: According to Google, Composer creates a custom image for the environment and passes one to each node, and if that got corrupted during scaling then the only way to fix it is by adding a new python package so it rebuilds the image. Incidentally, version 1.3.0 of Composer is much better as the scheduler is restarted every 10 minutes which should solve some of the latter issues I experienced.
answered Nov 14 '18 at 17:42
KJSKJS
426
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
@TrevorEdwards Yeah I did, I'm using thecomposer-1.1.1-airflow-1.9.0image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.
– KJS
Nov 14 '18 at 20:54
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53291305%2fcloud-composer-throwing-invalidtoken-after-adding-another-node%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I managed to fix this problem by adding a new python package. It seems that adding a package is the only way to really "reboot" the environment. The reboot invalidated all of my variables and connections when it had finished but I was able to just add those back in rather than having to recreate the entire environment.
Heard back about this issue: According to Google, Composer creates a custom image for the environment and passes one to each node, and if that got corrupted during scaling then the only way to fix it is by adding a new python package so it rebuilds the image. Incidentally, version 1.3.0 of Composer is much better as the scheduler is restarted every 10 minutes which should solve some of the latter issues I experienced.
answered Nov 14 '18 at 17:42
KJSKJS
426
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
@TrevorEdwards Yeah I did, I'm using thecomposer-1.1.1-airflow-1.9.0image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.
– KJS
Nov 14 '18 at 20:54
add a comment |
I managed to fix this problem by adding a new python package. It seems that adding a package is the only way to really "reboot" the environment. The reboot invalidated all of my variables and connections when it had finished but I was able to just add those back in rather than having to recreate the entire environment.
Heard back about this issue: According to Google, Composer creates a custom image for the environment and passes one to each node, and if that got corrupted during scaling then the only way to fix it is by adding a new python package so it rebuilds the image. Incidentally, version 1.3.0 of Composer is much better as the scheduler is restarted every 10 minutes which should solve some of the latter issues I experienced.
answered Nov 14 '18 at 17:42
KJSKJS
426
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
@TrevorEdwards Yeah I did, I'm using thecomposer-1.1.1-airflow-1.9.0image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.
– KJS
Nov 14 '18 at 20:54
add a comment |
I managed to fix this problem by adding a new python package. It seems that adding a package is the only way to really "reboot" the environment. The reboot invalidated all of my variables and connections when it had finished but I was able to just add those back in rather than having to recreate the entire environment.
Heard back about this issue: According to Google, Composer creates a custom image for the environment and passes one to each node, and if that got corrupted during scaling then the only way to fix it is by adding a new python package so it rebuilds the image. Incidentally, version 1.3.0 of Composer is much better as the scheduler is restarted every 10 minutes which should solve some of the latter issues I experienced.
answered Nov 14 '18 at 17:42
KJSKJS
426
I managed to fix this problem by adding a new python package. It seems that adding a package is the only way to really "reboot" the environment. The reboot invalidated all of my variables and connections when it had finished but I was able to just add those back in rather than having to recreate the entire environment.
Heard back about this issue: According to Google, Composer creates a custom image for the environment and passes one to each node, and if that got corrupted during scaling then the only way to fix it is by adding a new python package so it rebuilds the image. Incidentally, version 1.3.0 of Composer is much better as the scheduler is restarted every 10 minutes which should solve some of the latter issues I experienced.
answered Nov 14 '18 at 17:42
KJSKJS
426
edited Nov 16 '18 at 23:53
answered Nov 14 '18 at 17:42
KJSKJS
426
answered Nov 14 '18 at 17:42
KJSKJS
426
answered Nov 14 '18 at 17:42
KJSKJS
426
426
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
@TrevorEdwards Yeah I did, I'm using thecomposer-1.1.1-airflow-1.9.0image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.
– KJS
Nov 14 '18 at 20:54
add a comment |
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
@TrevorEdwards Yeah I did, I'm using thecomposer-1.1.1-airflow-1.9.0image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.
– KJS
Nov 14 '18 at 20:54
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
What version of Composer did you see this problem on? And did you attempt to override the fernet key manually (not recommended, just curious)?
– Trevor Edwards
Nov 14 '18 at 19:28
@TrevorEdwards Yeah I did, I'm using the
composer-1.1.1-airflow-1.9.0 image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.– KJS
Nov 14 '18 at 20:54
@TrevorEdwards Yeah I did, I'm using the
composer-1.1.1-airflow-1.9.0 image. It didn't work though, in Google's docs they mandate that it is permanently set upon environment creation and that they manage the key. Trying to override the configuration or the environment variable for the fernet key is denied.– KJS
Nov 14 '18 at 20:54
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53291305%2fcloud-composer-throwing-invalidtoken-after-adding-another-node%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I've never used Composer but is there a way to view the scheduler logs separate from the webserver, and if so does the error happen in both?
– joeb
Nov 14 '18 at 2:11
@joeb Stackdriver has the logs split for the scheduler, it unfortunately throws the same error as it tries to load the DAGs. Nothing has changed this morning, I think my best course of action is to recreate the environment.
– KJS
Nov 14 '18 at 16:48