Using the Twisted Web Client — Twisted 25.5.0 documentation

Using the Twisted Web Client — Twisted 25.5.0 documentation
Twisted Web
Developer Guides
Using the Twisted Web Client
View page source
Using the Twisted Web Client
Overview
This document describes how to use the HTTP client included in Twisted
Web. After reading it, you should be able to make HTTP and HTTPS
requests using Twisted Web. You will be able to specify the request
method, headers, and body and you will be able to retrieve the response
code, headers, and body.
A number of higher-level features are also explained, including proxying,
automatic content encoding negotiation, and cookie handling.
Prerequisites
This document assumes that you are familiar with
Deferreds and Failures
, and
producers and consumers
It also assumes you are familiar with the basic concepts of HTTP, such
as requests and responses, methods, headers, and message bodies. The
HTTPS section of this document also assumes you are somewhat familiar with
SSL and have read about
using SSL in Twisted
The Agent
Issuing Requests
The
twisted.web.client.Agent
class is the entry
point into the client API. Requests are issued using the
request
method, which
takes as parameters a request method, a request URI, the request headers,
and an object which can produce the request body (if there is to be one).
The agent is responsible for connection setup. Because of this, it
requires a reactor as an argument to its initializer. An example of
creating an agent and issuing a request using it might look like this:
request.py
from
twisted.internet
import
reactor
from
twisted.web.client
import
Agent
from
twisted.web.http_headers
import
Headers
agent
Agent
reactor
agent
request
"GET"
"http://httpbin.com/anything"
Headers
({
"User-Agent"
"Twisted Web Client Example"
]}),
None
def
cbResponse
ignored
):
"Response received"
addCallback
cbResponse
def
cbShutdown
ignored
):
reactor
stop
()
addBoth
cbShutdown
reactor
run
()
As may be obvious, this issues a new
GET
request for
to the web server on
example.com
Agent
is
responsible for resolving the hostname into an IP address and connecting
to it on port 80 (for
HTTP
URIs), port 443 (for
HTTPS
URIs), or on the port number specified in the URI itself. It is also
responsible for cleaning up the connection afterwards. This code sends
a request which includes one custom header,
User-Agent
. The
last argument passed to
Agent.request
is
None
though, so the request has no body.
Sending a request which does include a body requires passing an object
providing
twisted.web.iweb.IBodyProducer
to
Agent.request
. This interface extends the more general
IPushProducer
by adding a new
length
attribute and adding several
constraints to the way the producer and consumer interact.
The length attribute must be a non-negative integer or the constant
twisted.web.iweb.UNKNOWN_LENGTH
. If the length is known,
it will be used to specify the value for the
Content-Length
header in the request. If the length is
unknown the attribute should be set to
UNKNOWN_LENGTH
Since more servers support
Content-Length
, if a length can be
provided it should be.
An additional method is required on
IBodyProducer
implementations:
startProducing
. This method is used to
associate a consumer with the producer. It should return a
Deferred
which fires when all data has been produced.
IBodyProducer
implementations should never call the
consumer’s
unregisterProducer
method. Instead, when it
has produced all of the data it is going to produce, it should only
fire the
Deferred
returned by
startProducing
For additional details about the requirements of
IBodyProducer
implementations, see
the API documentation.
Here’s a simple
IBodyProducer
implementation which
writes an in-memory string to the consumer:
bytesprod.py
from
zope.interface
import
implementer
from
twisted.internet.defer
import
succeed
from
twisted.web.iweb
import
IBodyProducer
@implementer
IBodyProducer
class
BytesProducer
def
__init__
self
body
):
self
body
body
self
length
len
body
def
startProducing
self
consumer
):
consumer
write
self
body
return
succeed
None
def
pauseProducing
self
):
pass
def
stopProducing
self
):
pass
This producer can be used to issue a request with a body:
sendbody.py
from
bytesprod
import
BytesProducer
from
twisted.internet
import
reactor
from
twisted.web.client
import
Agent
from
twisted.web.http_headers
import
Headers
agent
Agent
reactor
body
BytesProducer
"hello, world"
agent
request
"POST"
"http://httpbin.org/post"
Headers
"User-Agent"
"Twisted Web Client Example"
],
"Content-Type"
"text/x-greeting"
],
),
body
def
cbResponse
ignored
):
"Response received"
addCallback
cbResponse
def
cbShutdown
ignored
):
reactor
stop
()
addBoth
cbShutdown
reactor
run
()
If you want to upload a file or you just have some data in a string, you
don’t have to copy
StringProducer
though. Instead, you can
use
FileBodyProducer
This
IBodyProducer
implementation works with any file-like
object (so use it with a
StringIO
if your upload data is
already in memory as a string); the idea is the same
as
StringProducer
from the previous example, but with a
little extra code to only send data as fast as the server will take it.
filesendbody.py
from
io
import
BytesIO
from
twisted.internet
import
reactor
from
twisted.web.client
import
Agent
FileBodyProducer
from
twisted.web.http_headers
import
Headers
agent
Agent
reactor
body
FileBodyProducer
BytesIO
"hello, world"
))
agent
request
"GET"
"http://example.com/"
Headers
"User-Agent"
"Twisted Web Client Example"
],
"Content-Type"
"text/x-greeting"
],
),
body
def
cbResponse
ignored
):
"Response received"
addCallback
cbResponse
def
cbShutdown
ignored
):
reactor
stop
()
addBoth
cbShutdown
reactor
run
()
FileBodyProducer
closes the file when it no longer needs it.
If the connection or the request take too much time, you can cancel the
Deferred
returned by the
Agent.request
method.
This will abort the connection, and the
Deferred
will errback
with
CancelledError
Receiving Responses
So far, the examples have demonstrated how to issue a request. However,
they have ignored the response, except for showing that it is a
Deferred
which seems to fire when the response has been
received. Next we’ll cover what that response is and how to interpret
it.
Agent.request
, as with most
Deferred
-returning
APIs, can return a
Deferred
which fires with a
Failure
. If the request fails somehow, this will be
reflected with a failure. This may be due to a problem looking up the
host IP address, or it may be because the HTTP server is not accepting
connections, or it may be because of a problem parsing the response, or
any other problem which arises which prevents the response from being
received. It does
not
include responses with an error status.
If the request succeeds, though, the
Deferred
will fire with
Response
. This
happens as soon as all the response headers have been received. It
happens before any of the response body, if there is one, is processed.
The
Response
object has several attributes giving the
response information: its code, version, phrase, and headers, as well as
the length of the body to expect. In addition to these, the
Response
also contains a reference to the
request
that it is
a response to; one particularly useful attribute on the request is
absoluteURI
The absolute URI to which the request was made. The
Response
object has a method which makes the response body
available:
deliverBody
. Using the
attributes of the response object and this method, here’s an example
which displays part of the response to a request:
response.py
from
pprint
import
pformat
from
twisted.internet
import
reactor
from
twisted.internet.defer
import
Deferred
from
twisted.internet.protocol
import
Protocol
from
twisted.web.client
import
Agent
from
twisted.web.http_headers
import
Headers
class
BeginningPrinter
Protocol
):
def
__init__
self
finished
):
self
finished
finished
self
remaining
1024
10
def
dataReceived
self
bytes
):
if
self
remaining
display
bytes
[:
self
remaining
"Some data received:"
display
self
remaining
-=
len
display
def
connectionLost
self
reason
):
"Finished receiving body:"
reason
getErrorMessage
())
self
finished
callback
None
agent
Agent
reactor
agent
request
"GET"
"http://httpbin.com/anything/"
Headers
({
"User-Agent"
"Twisted Web Client Example"
]}),
None
def
cbRequest
response
):
"Response version:"
response
version
"Response code:"
response
code
"Response phrase:"
response
phrase
"Response headers:"
pformat
list
response
headers
getAllRawHeaders
())))
finished
Deferred
()
response
deliverBody
BeginningPrinter
finished
))
return
finished
addCallback
cbRequest
def
cbShutdown
ignored
):
reactor
stop
()
addBoth
cbShutdown
reactor
run
()
The
BeginningPrinter
protocol in this example is passed to
Response.deliverBody
and the response body is then delivered
to its
dataReceived
method as it arrives. When the body has
been completely delivered, the protocol’s
connectionLost
method is called. It is important to inspect the
Failure
passed to
connectionLost
. If the response body has been
completely received, the failure will wrap a
twisted.web.client.ResponseDone
exception. This
indicates that it is
known
that all data has been received. It
is also possible for the failure to wrap a
twisted.web.http.PotentialDataLoss
exception: this
indicates that the server framed the response such that there is no way
to know when the entire response body has been received. Only
HTTP/1.0 servers should behave this way. Finally, it is possible for
the exception to be of another type, indicating guaranteed data loss for
some reason (a lost connection, a memory error, etc).
Just as protocols associated with a TCP connection are given a transport,
so will be a protocol passed to
deliverBody
. Since it makes
no sense to write more data to the connection at this stage of the
request, though, the transport
only
provides
IPushProducer
. This allows the
protocol to control the flow of the response data: a call to the
transport’s
pauseProducing
method will pause delivery; a
later call to
resumeProducing
will resume it. If it is
decided that the rest of the response body is not desired,
stopProducing
can be used to stop delivery permanently;
after this, the protocol’s
connectionLost
method will be
called.
An important thing to keep in mind is that the body will only be read
from the connection after
Response.deliverBody
is called.
This also means that the connection will remain open until this is done
(and the body read). So, in general, any response with a body
must
have that body read using
deliverBody
. If the
application is not interested in the body, it should issue a
HEAD
request or use a protocol which immediately calls
stopProducing
on its transport.
If the body of the response isn’t going to be consumed incrementally, then
readBody
can be used to get the body as a byte-string.
This function returns a
Deferred
that fires with the body after the request has been completed; cancelling this
Deferred
will close the connection to the HTTP server immediately.
responseBody.py
from
pprint
import
pformat
from
sys
import
argv
from
twisted.internet.task
import
react
from
twisted.web.client
import
Agent
readBody
from
twisted.web.http_headers
import
Headers
def
cbRequest
response
):
"Response version:"
response
version
"Response code:"
response
code
"Response phrase:"
response
phrase
"Response headers:"
pformat
list
response
headers
getAllRawHeaders
())))
readBody
response
addCallback
cbBody
return
def
cbBody
body
):
"Response body:"
body
def
main
reactor
url
"http://httpbin.org/get"
):
agent
Agent
reactor
agent
request
"GET"
url
Headers
({
"User-Agent"
"Twisted Web Client Example"
]}),
None
addCallback
cbRequest
return
react
main
argv
:])
Customizing your HTTPS Configuration
Everything you’ve read so far applies whether the scheme of the request URI is
HTTP
or
HTTPS
Since version 15.0.0, Twisted’s
Agent
HTTP client will validate
https
URLs against your platform’s trust store by default.
Note
If you’re using an earlier version, please upgrade, as this is a
critical
security feature!
Note
Only
Agent
, and things that use it, validates HTTPS.
Do not use
getPage
to retrieve HTTPS URLs; although we have not yet removed all the examples that use it, we discourage its use.
You should
pip
install
twisted[tls]
in order to get all the dependencies necessary to do TLS properly.
The
installation instructions
gives more detail on optional dependencies and how to install them which may be of interest.
For some uses, you may need to customize Agent’s use of HTTPS; for example, to provide a client certificate, or to use a custom certificate authority for an internal network.
Here, we’re just going to show you how to inject the relevant TLS configuration into an Agent, so we’ll use the simplest possible example, rather than a more useful but complex one.
Agent
’s constructor takes an optional second argument, which allows you to customize its behavior with respect to HTTPS.
The object passed here must provide the
twisted.web.iweb.IPolicyForHTTPS
interface.
Using the very helpful
badssl.com
web API, we will construct a request that fails to validate, because the certificate has the wrong hostname in it.
The following Python program should produce a verification error when run as
python
example.py
import
sys
from
twisted.internet.task
import
react
from
twisted.web.client
import
Agent
ResponseFailed
@react
def
main
reactor
):
agent
Agent
reactor
requested
agent
request
"GET"
sys
argv
encode
"ascii"
))
def
gotResponse
response
):
response
code
def
noResponse
failure
):
failure
trap
ResponseFailed
failure
value
reasons
getTraceback
())
return
requested
addCallbacks
gotResponse
noResponse
The verification error you see when running the above program is due to the mismatch between “
wrong.host.badssl.com
”, the expected hostname derived from the URL, and “
badssl.com
”, the hostname contained in the certificate presented by the server.
In our pretend scenario, we want to construct an
Agent
that validates HTTPS certificates normally, _except_ for this one host.
For
"wrong.host.badssl.com"
, we wish to supply the correct hostname to check the certificate against manually, to work around this problem.
In order to do that, we will supply our own policy that creates a different TLS configuration depending on the hostname, like so:
import
sys
from
zope.interface
import
implementer
from
twisted.internet.task
import
react
from
twisted.internet.ssl
import
optionsForClientTLS
from
twisted.web.iweb
import
IPolicyForHTTPS
from
twisted.web.client
import
Agent
ResponseFailed
BrowserLikePolicyForHTTPS
@implementer
IPolicyForHTTPS
class
OneHostnameWorkaroundPolicy
object
):
def
__init__
self
):
self
_normalPolicy
BrowserLikePolicyForHTTPS
()
def
creatorForNetloc
self
hostname
port
):
if
hostname
==
"wrong.host.badssl.com"
hostname
"badssl.com"
return
self
_normalPolicy
creatorForNetloc
hostname
port
@react
def
main
reactor
):
agent
Agent
reactor
OneHostnameWorkaroundPolicy
())
requested
agent
request
"GET"
sys
argv
encode
"ascii"
))
def
gotResponse
response
):
response
code
def
noResponse
failure
):
failure
trap
ResponseFailed
failure
value
reasons
getTraceback
())
return
requested
addCallbacks
gotResponse
noResponse
Now, invoking
python
example.py
will happily give us a
200
status code; however, running it with
or
or any of the other error hostnames should still give an error.
Note
The technique presented above does not
ignore the mismatching hostname
, but rather, it
provides the erroneous hostname specifically
so that the misconfiguration is expected.
Twisted does not document any facilities for disabling verification, since that makes TLS useless; instead, we strongly encourage you to figure out what properties you need from the connection and verify those.
Using this TLS policy mechanism, you can customize Agent to use any feature of TLS that Twisted has support for, including the examples given above; client certificates, alternate trust roots, and so on.
For a more detailed explanation of what options exist for client TLS configuration in Twisted, check out the documentation for the
optionsForClientTLS
API and the
Using SSL in Twisted
chapter of this documentation.
HTTP Persistent Connection
HTTP persistent connections use the same TCP connection to send and
receive multiple HTTP requests/responses. This reduces latency and TCP
connection establishment overhead.
The constructor of
twisted.web.client.Agent
takes an optional parameter pool, which should be an instance
of
HTTPConnectionPool
, which will be used
to manage the connections. If the pool is created with the
parameter
persistent
set to
True
(the
default), it will not close connections when the request is done, and
instead hold them in its cache to be re-used.
Here’s an example which sends requests over a persistent connection:
from
twisted.internet
import
reactor
from
twisted.internet.defer
import
Deferred
DeferredList
from
twisted.internet.protocol
import
Protocol
from
twisted.web.client
import
Agent
HTTPConnectionPool
class
IgnoreBody
Protocol
):
def
__init__
self
deferred
):
self
deferred
deferred
def
dataReceived
self
bytes
):
pass
def
connectionLost
self
reason
):
self
deferred
callback
None
def
cbRequest
response
):
'Response code:'
response
code
finished
Deferred
()
response
deliverBody
IgnoreBody
finished
))
return
finished
pool
HTTPConnectionPool
reactor
agent
Agent
reactor
pool
pool
def
requestGet
url
):
agent
request
'GET'
url
addCallback
cbRequest
return
# Two requests to the same host:
requestGet
'http://localhost:8080/foo'
addCallback
lambda
ign
requestGet
"http://localhost:8080/bar"
))
def
cbShutdown
ignored
):
reactor
stop
()
addCallback
cbShutdown
reactor
run
()
Here, the two requests are to the same host, one after the each
other. In most cases, the same connection will be used for the second
request, instead of two different connections when using a
non-persistent pool.
Multiple Connections to the Same Server
twisted.web.client.HTTPConnectionPool
instances
have an attribute
called
maxPersistentPerHost
which limits the
number of cached persistent connections to the same server. The default
value is 2. This is effective only when the
persistent
option is
True. You can change the value like bellow:
from
twisted.web.client
import
HTTPConnectionPool
pool
HTTPConnectionPool
reactor
persistent
True
pool
maxPersistentPerHost
With the default value of 2, the pool keeps around two connections to
the same host at most. Eventually the cached persistent connections will
be closed, by default after 240 seconds; you can change this timeout
value with the
cachedConnectionTimeout
attribute of the pool. To force all connections to close use
the
closeCachedConnections
method.
Automatic Retries
If a request fails without getting a response, and the request is
something that hopefully can be retried without having any side-effects
(e.g. a request with method GET), it will be retried automatically when
sending a request over a previously-cached persistent connection. You can
disable this behavior by setting
retryAutomatically
to
False
. Note that each request will only be retried
once.
Following redirects
By itself,
Agent
doesn’t follow HTTP redirects (responses
with 301, 302, 303, 307 status codes and a
location
header
field). You need to use the
twisted.web.client.RedirectAgent
class to do so. It
implements a rather strict behavior of the RFC, meaning it will redirect
301 and 302 as 307, only on
GET
and
HEAD
requests.
The following example shows how to have a redirect-enabled agent.
from
twisted.python.log
import
err
from
twisted.web.client
import
Agent
RedirectAgent
from
twisted.internet
import
reactor
def
display
response
):
"Received response"
response
def
main
():
agent
RedirectAgent
Agent
reactor
))
agent
request
"GET"
"http://example.com/"
addCallbacks
display
err
addCallback
lambda
ignored
reactor
stop
())
reactor
run
()
if
__name__
==
"__main__"
main
()
In contrast,
twisted.web.client.BrowserLikeRedirectAgent
implements
more lenient behaviour that closely emulates what web browsers do; in
other words 301 and 302
POST
redirects are treated like 303,
meaning the method is changed to
GET
before making the redirect
request.
As mentioned previously,
Response
contains a reference to both
the
request
that it is a response
to, and the previously received
response
, accessible by
previousResponse
In most cases there will not be a previous response, but in the case of
RedirectAgent
the response history can be obtained by
following the previous responses from response to response.
Using a HTTP proxy
To be able to use HTTP proxies with an agent, you can use the
twisted.web.client.ProxyAgent
class.
It supports the same interface as
Agent
, but takes the endpoint of the proxy as initializer argument.
This is specifically intended for talking to servers that implement the proxying variation of the HTTP protocol; for other types of proxies you will want
Agent.usingEndpointFactory
(see documentation below).
Here’s an example demonstrating the use of an HTTP proxy running on
localhost:8000.
from
twisted.python.log
import
err
from
twisted.web.client
import
ProxyAgent
from
twisted.internet
import
reactor
from
twisted.internet.endpoints
import
TCP4ClientEndpoint
def
display
response
):
"Received response"
response
def
main
():
endpoint
TCP4ClientEndpoint
reactor
"localhost"
8000
agent
ProxyAgent
endpoint
agent
request
"GET"
"https://example.com/"
addCallbacks
display
err
addCallback
lambda
ignored
reactor
stop
())
reactor
run
()
if
__name__
==
"__main__"
main
()
Please refer to the
endpoints documentation
for
more information about how they work and the
twisted.internet.endpoints
API documentation to learn
what other kinds of endpoints exist.
Handling HTTP cookies
An existing agent instance can be wrapped with
twisted.web.client.CookieAgent
to automatically store, send and
track HTTP cookies. A
CookieJar
instance, from the Python standard library
module
http.cookiejar
is used to store the cookie information. An example of using
CookieAgent
to
perform a request and display the collected cookies might look like this:
cookies.py
from
http.cookiejar
import
CookieJar
from
twisted.internet
import
reactor
from
twisted.python
import
log
from
twisted.web.client
import
Agent
CookieAgent
def
displayCookies
response
cookieJar
):
"Received response"
response
"Cookies:"
len
cookieJar
))
for
in
cookieJar
def
main
():
cookieJar
CookieJar
()
agent
CookieAgent
Agent
reactor
),
cookieJar
agent
request
"GET"
"http://httpbin.org/cookies/set?some=data"
addCallback
displayCookies
cookieJar
addErrback
log
err
addCallback
lambda
ignored
reactor
stop
())
reactor
run
()
if
__name__
==
"__main__"
main
()
Automatic Content Encoding Negotiation
twisted.web.client.ContentDecoderAgent
adds
support for sending
Accept-Encoding
request headers and
interpreting
Content-Encoding
response headers. These headers
allow the server to encode the response body somehow, typically with some
compression scheme to save on transfer
costs.
ContentDecoderAgent
provides this functionality as a
wrapper around an existing agent instance. Together with one or more
decoder objects (such as
twisted.web.client.GzipDecoder
), this wrapper
automatically negotiates an encoding to use and decodes the response body
accordingly. To application code using such an agent, there is no visible
difference in the data delivered.
gzipdecoder.py
from
twisted.internet
import
reactor
from
twisted.internet.defer
import
Deferred
from
twisted.internet.protocol
import
Protocol
from
twisted.python
import
log
from
twisted.web.client
import
Agent
ContentDecoderAgent
GzipDecoder
class
BeginningPrinter
Protocol
):
def
__init__
self
finished
):
self
finished
finished
self
remaining
1024
10
def
dataReceived
self
bytes
):
if
self
remaining
display
bytes
[:
self
remaining
"Some data received:"
display
self
remaining
-=
len
display
def
connectionLost
self
reason
):
"Finished receiving body:"
reason
type
reason
value
self
finished
callback
None
def
printBody
response
):
finished
Deferred
()
response
deliverBody
BeginningPrinter
finished
))
return
finished
def
main
():
agent
ContentDecoderAgent
Agent
reactor
),
[(
"gzip"
GzipDecoder
)])
agent
request
"GET"
"http://httpbin.org/gzip"
addCallback
printBody
addErrback
log
err
addCallback
lambda
ignored
reactor
stop
())
reactor
run
()
if
__name__
==
"__main__"
main
()
Implementing support for new content encodings is as simple as writing a
new class like
GzipDecoder
that can decode a response using
the new encoding. As there are not many content encodings in widespread
use, gzip is the only encoding supported by Twisted itself.
Connecting To Non-standard Destinations
Typically you want your HTTP client to open a TCP connection directly to the web server.
Sometimes however it’s useful to be able to connect some other way, e.g. making an HTTP request over a SOCKS proxy connection or connecting to a server listening on a UNIX socket.
For this reason, there is an alternate constructor called
Agent.usingEndpointFactory
that takes an
endpointFactory
argument.
This argument must provide the
twisted.web.iweb.IAgentEndpointFactory
interface.
Note that when talking to a HTTP proxy, i.e. a server that implements the proxying-specific variant of HTTP you should use
ProxyAgent
- see documentation above.
endpointconstructor.py
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
"""
Send a HTTP request to Docker over a Unix socket.
Will probably need to be run as root.
Usage:
$ sudo python endpointconstructor.py []
"""
from
sys
import
argv
from
zope.interface
import
implementer
from
twisted.internet.endpoints
import
UNIXClientEndpoint
from
twisted.internet.task
import
react
from
twisted.web.client
import
Agent
readBody
from
twisted.web.iweb
import
IAgentEndpointFactory
@implementer
IAgentEndpointFactory
class
DockerEndpointFactory
"""
Connect to Docker's Unix socket.
"""
def
__init__
self
reactor
):
self
reactor
reactor
def
endpointForURI
self
uri
):
return
UNIXClientEndpoint
self
reactor
"/var/run/docker.sock"
def
main
reactor
path
"/containers/json?all=1"
):
agent
Agent
usingEndpointFactory
reactor
DockerEndpointFactory
reactor
))
agent
request
"GET"
"unix://localhost"
path
addCallback
readBody
addCallback
return
react
main
argv
:])
Conclusion
You should now understand the basics of the Twisted Web HTTP client. In
particular, you should understand:
How to issue requests with arbitrary methods, headers, and bodies.
How to access the response version, code, phrase, headers, and body.
How to store, send, and track cookies.
How to control the streaming of the response body.
How to enable the HTTP persistent connection, and control the
number of connections.