Speeding up blob copying on Azure
I have a use case which often requires to copy a blob (file) from one Azure region to another. The file size spans from 25 to 45GB. Needless to say, this sometimes goes very slowly, with inconsistent performance. This might take up to two hours, sometimes more. Distance plays a role, but it differs. Even within the same region copying is slower then I would expect. I've been trying:
- The Python SDK, and its copy blob method from the blob service.
- The rest API copy blob
- az copy from the CLI.
Although I didn't really expect different results, since all of them use the same backend methods.
Is there any approach I am missing? Is there any way to speed up this process, or any kind of blob sharing integrated in Azure? VHD/disk sharing could also do.
azure cloud azure-storage azure-sdk-python asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
add a comment |
I have a use case which often requires to copy a blob (file) from one Azure region to another. The file size spans from 25 to 45GB. Needless to say, this sometimes goes very slowly, with inconsistent performance. This might take up to two hours, sometimes more. Distance plays a role, but it differs. Even within the same region copying is slower then I would expect. I've been trying:
- The Python SDK, and its copy blob method from the blob service.
- The rest API copy blob
- az copy from the CLI.
Although I didn't really expect different results, since all of them use the same backend methods.
Is there any approach I am missing? Is there any way to speed up this process, or any kind of blob sharing integrated in Azure? VHD/disk sharing could also do.
azure cloud azure-storage azure-sdk-python asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
add a comment |
I have a use case which often requires to copy a blob (file) from one Azure region to another. The file size spans from 25 to 45GB. Needless to say, this sometimes goes very slowly, with inconsistent performance. This might take up to two hours, sometimes more. Distance plays a role, but it differs. Even within the same region copying is slower then I would expect. I've been trying:
- The Python SDK, and its copy blob method from the blob service.
- The rest API copy blob
- az copy from the CLI.
Although I didn't really expect different results, since all of them use the same backend methods.
Is there any approach I am missing? Is there any way to speed up this process, or any kind of blob sharing integrated in Azure? VHD/disk sharing could also do.
azure cloud azure-storage azure-sdk-python asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
I have a use case which often requires to copy a blob (file) from one Azure region to another. The file size spans from 25 to 45GB. Needless to say, this sometimes goes very slowly, with inconsistent performance. This might take up to two hours, sometimes more. Distance plays a role, but it differs. Even within the same region copying is slower then I would expect. I've been trying:
- The Python SDK, and its copy blob method from the blob service.
- The rest API copy blob
- az copy from the CLI.
Although I didn't really expect different results, since all of them use the same backend methods.
Is there any approach I am missing? Is there any way to speed up this process, or any kind of blob sharing integrated in Azure? VHD/disk sharing could also do.
azure cloud azure-storage azure-sdk-python azure cloud azure-storage azure-sdk-python asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
asked Nov 12 '18 at 18:53
Aleksandar Stojadinovic
2,7282140
2,7282140
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You may want to try /SyncCopy option in AzCopy:
Synchronously copy blobs from one storage account to another
AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob is copied, and AzCopy periodically checks the copy status until the copying is completed or failed.
The
/SyncCopyoption ensures that the copy operation gets consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory, and then uploading them to the Blob storage destination.
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy
/SyncCopymight generate additional egress cost compared to asynchronous copy, the recommended approach is to use this option in an Azure VM that is in the same region as your source storage account to avoid egress cost.
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53268375%2fspeeding-up-blob-copying-on-azure%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You may want to try /SyncCopy option in AzCopy:
Synchronously copy blobs from one storage account to another
AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob is copied, and AzCopy periodically checks the copy status until the copying is completed or failed.
The
/SyncCopyoption ensures that the copy operation gets consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory, and then uploading them to the Blob storage destination.
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy
/SyncCopymight generate additional egress cost compared to asynchronous copy, the recommended approach is to use this option in an Azure VM that is in the same region as your source storage account to avoid egress cost.
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
add a comment |
You may want to try /SyncCopy option in AzCopy:
Synchronously copy blobs from one storage account to another
AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob is copied, and AzCopy periodically checks the copy status until the copying is completed or failed.
The
/SyncCopyoption ensures that the copy operation gets consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory, and then uploading them to the Blob storage destination.
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy
/SyncCopymight generate additional egress cost compared to asynchronous copy, the recommended approach is to use this option in an Azure VM that is in the same region as your source storage account to avoid egress cost.
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
add a comment |
You may want to try /SyncCopy option in AzCopy:
Synchronously copy blobs from one storage account to another
AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob is copied, and AzCopy periodically checks the copy status until the copying is completed or failed.
The
/SyncCopyoption ensures that the copy operation gets consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory, and then uploading them to the Blob storage destination.
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy
/SyncCopymight generate additional egress cost compared to asynchronous copy, the recommended approach is to use this option in an Azure VM that is in the same region as your source storage account to avoid egress cost.
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
You may want to try /SyncCopy option in AzCopy:
Synchronously copy blobs from one storage account to another
AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob is copied, and AzCopy periodically checks the copy status until the copying is completed or failed.
The
/SyncCopyoption ensures that the copy operation gets consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory, and then uploading them to the Blob storage destination.
AzCopy /Source:https://myaccount1.blob.core.windows.net/myContainer/ /Dest:https://myaccount2.blob.core.windows.net/myContainer/ /SourceKey:key1 /DestKey:key2 /Pattern:ab /SyncCopy
/SyncCopymight generate additional egress cost compared to asynchronous copy, the recommended approach is to use this option in an Azure VM that is in the same region as your source storage account to avoid egress cost.
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
answered Nov 13 '18 at 2:54
Zhaoxing Lu - Microsoft
3,398621
3,398621
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
add a comment |
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
Ok, I'll try that out, although I've seen that and didn't correlate it to better performance. If I understand well, the blob will be downloaded to the invoking machine, and then re-uploaded to another location? Shouldn't that be slower?
– Aleksandar Stojadinovic
Nov 13 '18 at 8:56
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
It can have consistent speed, either slower or faster is possible. The recommended approach is to use this option in an Azure VM that is in the same region as your source storage account.
– Zhaoxing Lu - Microsoft
Nov 13 '18 at 11:02
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53268375%2fspeeding-up-blob-copying-on-azure%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
