Locust: The pycurl client

[likezjuisee]

I want to use pycurl to do the http request, so I make a pycurl client likes below:

coding=utf-8

from locust import Locust, events
import time
import pycurl
from cStringIO import StringIO
from exceptions import AttributeError

class PycurlClient(object):

def __init__(self):
    self.curl = None
    self.recordFuncs = ['perform']

def openCurl(self):
    self.curl = pycurl.Curl()

def closeCurl(self):
    try:
        self.curl.close()
    except:
        pass

def __getattr__(self, name):
    try:
        func = eval('self.curl.' + name)
    except AttributeError:
        print 'pycurl does not have the method %s' % name

    def wrapper(*args, **kwargs):
        start_time = time.time()
        try:
            result = func(*args, **kwargs)
        except Exception as e:
            total_time = int((time.time() - start_time) * 1000)
            events.request_failure.fire(request_type="pycurl", name=name, response_time=total_time, exception=e)
        else:
            if name in self.recordFuncs:
                total_time = int((time.time() - start_time) * 1000)
                events.request_success.fire(request_type="pycurl", name=name, response_time=total_time, response_length=0)
    return wrapper

class PycurlLocust(Locust):

client = None

def __init__(self, *args, **kwargs):
    super(PycurlLocust, self).__init__(*args, **kwargs)
    self.client = PycurlClient()

And, I use this client in my new task like this:

coding=utf-8

import pycurl
from locust.pycurlclients import PycurlClient, PycurlLocust
from locust import TaskSet, task
from cStringIO import StringIO

class TestTaskSet(TaskSet):
@task
def index(self):
print '%s get index' % str(self.locust)
self.client.openCurl()
self.client.setopt(pycurl.URL, r'http://www.so.com')
sio = StringIO()
self.client.setopt(pycurl.WRITEFUNCTION, sio.write)
self.client.perform()
self.client.closeCurl()

def on_start(self):
    print 'locust %s' % str(self.locust)

class TestPycurlClient(PycurlLocust):
max_wait = min_wait = 1
task_set = TestTaskSet
host = r'http://www.so.com'

When the user was 1, the pycurl's performance was better than requests, but when I increased the user's number, the requests number of per second equals to the 1 user almostly. So do you have some suggestions about this?

[cgoldberg]

I don't have an answer to your question, but I'm very confused why you would replace requests with pycurl. Can you elaborate on what you are trying to achieve with this code?

[likezjuisee]

requests lib is a sync net io lib, so do you have a plan to change it to async lib, maybe python3.5's asyncio.

[likezjuisee]

And httptools is a nice http response parser, it might speed up the io throughout.

[heyman]

@likezjuisee: Locust uses gevent which monkey patches python sockets and makes them use asynchronous IO (while maintaining a synchronous programming model).

Python-requests gives you an extremely nice API and handles a lot of edge-cases automatically. Therefore it comes with some overhead. Therefore, if your planning to run huge load tests you might benefit from using an HTTP client with less overhead. In that case I would recommend using geventhttpclient (https://github.com/gwik/geventhttpclient).

I don't know about PyCurl but I suspect that it's synchronous by default, and since it's written in C, it bypasses gevents monkey patching of socket which will make it run synchronously, which would explain why you would get the same result as running a single user (or worse). Also, I wouldn't expect eval() to be performant, so instead of overriding __getattr__ it would be probably better to make a class/subclass with methods that "manually" call the functions that makes the requests.

[likezjuisee]

I known the monkey patches, but I guess it may be worse than asyncio. You might look around the web server framework sanic https://github.com/channelcat/sanic, it benefits from the httptools and uvloop. So I supposed you to update the locust to python3.5. If you persist with your own plan, it is ok. ^_^

[gautamdivgi]

I came upon this issue and know its closed. However, I just wanted to point out one thing. The reason I was looking to use pycurl is to get internal metrics on the actual HTTP call. The "response time" for an call will include multiple things - DNS lookup, TLS negotiation, server latency to process the call, etc. Curl makes those available. That's the reason I was looking to use pycurl. Are those metrics provided via locust today? I didn't see them but thought I'd ask. For reference here's what curl provides for metrics - https://ec.haxx.se/usingcurl-writeout.html