SPOILER ALERT!
The way to download and upload major file breakpoints on OSS.txt
Text / Umbrella Xing (Supply: Alibaba Cloud official site)
OSS (Open Storage Service) is often a storage service for massive unstructured information objects. Together with the recognition and fast growth of cloud computing, increasingly more developers are building their applications on major of OSS, but you will find not lots of development supplies for OSS. I am right here to assist others, by answering a often asked question-how to attain the breakpoint resume function of huge files on OSS-I hope that more masters can share their development experience on cloud storage.
OSS provides an external interface in the kind of RESTful. Certainly one of its most significant attributes is statelessness, that's, the OSS server does not preserve the communication state of any client that communicates with it except for a single request. Consequently, how to recover raw disk for the realization of stateful functions including resuming of breakpoints is how you can comprehensive state maintenance around the client. Inside the following chapters, I use Python as an instance to introduce how you can implement OSS breakpoint download and breakpoint upload.
Breakpoint download of substantial files on OSS
The so-called breakpoint download is to continue downloading from the location where the file has been downloaded. To facilitate understanding, let's 1st appear at a Python instance that downloads a file from OSS and saves it locally. Within this example [Note: So that you can facilitate understanding, the code examples within this article ignore some simple error handling and extreme case judgment logic], we download a file called 'example.dat' from a bucket named 'lingyun' And save it within the existing directory together with the identical name.
Primarily based on the above code, the following system shows the file download code that adds the breakpoint resume function, as well as the changed place is marked with green color:
Compared with all the previous code, you can find four adjustments within this code:
Added logic for streaming to neighborhood files. Protect against downloaded information objects from getting as well massive to be read into regional memory at after. Prior to sending information to OSS, get the neighborhood file length. Construct an HTTP RangeHeader and ask the OSS to start downloading from the specified place. Judge the HTTP worth returned by OSS, and make corresponding processing: If OSS returns 206, it implies that the information downloaded is within the specified location variety; other status codes indicate that the 'Range' parameter is incorrect or abnormal. When making use of the HTTP parameter 'Range', please note the following three points:
The file position within the Range parameter begins from 0, plus the maximum value is the file length minus 1. When the Variety parameter is filled in incorrectly, OSS will ignore this parameter. [Note: When data recovery from raw usb or hard drive are legal, this request will conform to the syntax of the getobject request, and OSS will return The content material on the complete object, not element from the data expected by the user. ]. In the event the Variety parameter is set properly, OSS will return HTTP status code 206 (not 200) to indicate that portion with the data is returned via the 'Range' parameter, and concurrent download of large files also can be achieved. This function is left as a query for readers, interested readers can implement it by themselves. The OSS official SDK also gives a multi-threaded download function for the reference.
Implement breakpoint upload of substantial files on OSS
In comparison to breakpoint download, the implementation of breakpoint upload is certainly considerably more difficult. The resolution offered by OSS may be understood as: dividing the large file into several tiny information blocks appropriate for public network transmission around the client; then uploading these small information blocks towards the OSS; lastly, merging these modest data blocks on the OSS server In to the final document. So that you can achieve this function, OSS separately released a set of upload API interface-MultipartUpload. There are actually 6 API interfaces in total:
InitiateMultipartUpload: Initialize a MultipartUpload event UploadPart: Upload a data block CompleteMultipartUpload: Comprehensive a MultipartUpload occasion AbortMultipartUpload: Abort a MultipartUpload event ListMultipartUploads: List all current MultipartUpload events ListParts: List all information blocks beneath a MultipartUpload event Defined in this set of interfaces Two exclusive identification codes (UUIDs): UploadID and PartID, are employed to determine a MultipartUpload upload occasion plus a information block, respectively. A total Multipart upload approach consists in the following steps:
1)InitiateMultipartUpload: Initialize a MultipartUpload event
The client informs OSS to upload a large file, and OSS returns to the client an UploadID that uniquely identifies this Multipart upload occasion. Python sample code is as follows:
The following is an instance in the HTTP result returned by OSS:
'0004D4184129F5A1A42663160C4C58B1' would be the UploadID assigned by OSS for this MultipartUpload event. By means of this interface, the user just registered a MultipartUpload occasion around the OSS, and no file was made or changed. You can build numerous MultipartUpload events on the same file. %8 of these MultipartUpload events are not completed (Comprehensive) or aborted (Abort)
Tags: cloud computing, how Alibaba Cloud implements breakpoint download and upload of big files on OSS
OSS (Open Storage Service) is often a storage service for massive unstructured information objects. Together with the recognition and fast growth of cloud computing, increasingly more developers are building their applications on major of OSS, but you will find not lots of development supplies for OSS. I am right here to assist others, by answering a often asked question-how to attain the breakpoint resume function of huge files on OSS-I hope that more masters can share their development experience on cloud storage.
OSS provides an external interface in the kind of RESTful. Certainly one of its most significant attributes is statelessness, that's, the OSS server does not preserve the communication state of any client that communicates with it except for a single request. Consequently, how to recover raw disk for the realization of stateful functions including resuming of breakpoints is how you can comprehensive state maintenance around the client. Inside the following chapters, I use Python as an instance to introduce how you can implement OSS breakpoint download and breakpoint upload.
Breakpoint download of substantial files on OSS
The so-called breakpoint download is to continue downloading from the location where the file has been downloaded. To facilitate understanding, let's 1st appear at a Python instance that downloads a file from OSS and saves it locally. Within this example [Note: So that you can facilitate understanding, the code examples within this article ignore some simple error handling and extreme case judgment logic], we download a file called 'example.dat' from a bucket named 'lingyun' And save it within the existing directory together with the identical name.
Primarily based on the above code, the following system shows the file download code that adds the breakpoint resume function, as well as the changed place is marked with green color:
Compared with all the previous code, you can find four adjustments within this code:
Added logic for streaming to neighborhood files. Protect against downloaded information objects from getting as well massive to be read into regional memory at after. Prior to sending information to OSS, get the neighborhood file length. Construct an HTTP RangeHeader and ask the OSS to start downloading from the specified place. Judge the HTTP worth returned by OSS, and make corresponding processing: If OSS returns 206, it implies that the information downloaded is within the specified location variety; other status codes indicate that the 'Range' parameter is incorrect or abnormal. When making use of the HTTP parameter 'Range', please note the following three points:
The file position within the Range parameter begins from 0, plus the maximum value is the file length minus 1. When the Variety parameter is filled in incorrectly, OSS will ignore this parameter. [Note: When data recovery from raw usb or hard drive are legal, this request will conform to the syntax of the getobject request, and OSS will return The content material on the complete object, not element from the data expected by the user. ]. In the event the Variety parameter is set properly, OSS will return HTTP status code 206 (not 200) to indicate that portion with the data is returned via the 'Range' parameter, and concurrent download of large files also can be achieved. This function is left as a query for readers, interested readers can implement it by themselves. The OSS official SDK also gives a multi-threaded download function for the reference.
Implement breakpoint upload of substantial files on OSS
In comparison to breakpoint download, the implementation of breakpoint upload is certainly considerably more difficult. The resolution offered by OSS may be understood as: dividing the large file into several tiny information blocks appropriate for public network transmission around the client; then uploading these small information blocks towards the OSS; lastly, merging these modest data blocks on the OSS server In to the final document. So that you can achieve this function, OSS separately released a set of upload API interface-MultipartUpload. There are actually 6 API interfaces in total:
InitiateMultipartUpload: Initialize a MultipartUpload event UploadPart: Upload a data block CompleteMultipartUpload: Comprehensive a MultipartUpload occasion AbortMultipartUpload: Abort a MultipartUpload event ListMultipartUploads: List all current MultipartUpload events ListParts: List all information blocks beneath a MultipartUpload event Defined in this set of interfaces Two exclusive identification codes (UUIDs): UploadID and PartID, are employed to determine a MultipartUpload upload occasion plus a information block, respectively. A total Multipart upload approach consists in the following steps:
1)InitiateMultipartUpload: Initialize a MultipartUpload event
The client informs OSS to upload a large file, and OSS returns to the client an UploadID that uniquely identifies this Multipart upload occasion. Python sample code is as follows:
The following is an instance in the HTTP result returned by OSS:
'0004D4184129F5A1A42663160C4C58B1' would be the UploadID assigned by OSS for this MultipartUpload event. By means of this interface, the user just registered a MultipartUpload occasion around the OSS, and no file was made or changed. You can build numerous MultipartUpload events on the same file. %8 of these MultipartUpload events are not completed (Comprehensive) or aborted (Abort)
Tags: cloud computing, how Alibaba Cloud implements breakpoint download and upload of big files on OSS
