Body is a string in some encoding. In Python 2, we can print it without knowing what the encoding is. PycURL does not provide storage for the network response – that is the replicate binary option’s job.
PycURL to write to that buffer. While the WRITEFUNCTION idiom continues to work, it is now unnecessary. In Python 3, PycURL response the response body as a byte string. This is handy if we are downloading a binary file, but for text documents we must decode the byte string. In the above example, we assume that the body is encoded in iso-8859-1. Python 2 and Python 3 versions can be combined. Doing so requires decoding the response body as in Python 3 version.
In order to take advantage of security that HTTPS provides, PycURL needs to utilize a certificate bundle. HTTP standard specifies that headers are encoded in iso-8859-1. On Python 2, decoding step can be skipped. On Python 3, decoding step is required. We are going to ignore all lines that don’t have a colon in them. Break the header line into header name and value.
Remove whitespace that may be present. Now we can actually record the header name and value. Note: this only works when headers are not duplicated, see below. Figure out what encoding was sent with the response, if any. Default encoding for HTML is iso-8859-1. Decode using the encoding we figured out. That was a lot of code for something very straightforward.
Unfortunately, as libcurl refrains from allocating memory for response data, it is on our application to perform this grunt work. One caveat with the above code is that if there are multiple headers for the same name, such as Set-Cookie, only the last header value will be stored. Suppose we want to save response body to a file. The important part is opening the file in binary mode – then response body can be written bytewise without decoding or encoding steps. By default libcurl, and PycURL, do not follow redirects. As we did not set a write callback, the default libcurl and PycURL behavior to write response body to standard output takes effect. Following redirects is one option that libcurl provides.