I know it’s too much too hope that I have any regular readers, but if you come here even infrequently you may have noticed it’s been seven months since I last posted. You may even have wondered what I have been up to. What could’ve possible kept Tony so busy that he would totally ignore his blog for more than half a year? Well, I’ll tell you. Fortnite. I have been busy getting my ass kicked 24-7 by eleven year-olds in Fortnite. It is embarrassing quite honestly.
In between matches, however, I actually have been productive working on intelligent agents for trading cryptocurrency on the Coinbase Pro (formerly GDAX) exchange. While most of the high level code isn’t ready for prime time yet, a good portion of the low level, workhorse code has been road tested and is relatively stable. Since other developers might find it useful, I’ve decided to begin releasing it into the wild.
This week I released the first of my code: CoPrA, an asyncronous Python WebSocket client. CoPrA is built on top of the phenomenal Autobahn|Python WebSocket framework. While it is certainly possible to use Autobahn itself for a Coinbase client, its options are multitudinous and generally more complicated than necessary for most applications. My goal with CoPrA, therefore, was to simplify the WebSocket interface without sacrificing any of the functionality I need.
I will be following up shortly with a more detailed tutorial for using CoPrA, but if you’d like to take it for a test drive now, you can install it using pip:
$ pip3 install copra
CoPrA can also be installed from source. Either clone the repository from github:
$ git clone https://github.com/tpodlaski/copra
or download the tarball:
$ curl -OL https://github.com/tpodlaski/copra/tarball/master'
Once you’ve downloaded a copy of the source files, you can install CoPrA with:
$ python3 setup.py install
I’m looking for feedback, suggestions, bugs, or any other comments, so please feel free to leave them in the comments below. Look for another post on CoPrA soon.
Links:
- CoPrA Source Code: https://github.com/tpodlaski/copra
- CoPrA on PyPi: https://pypi.org/project/copra/
- CoPrA Documentation: https://copra.readthedocs.io/en/latest/
30 comments
Skip to comment form
Awesome… will try this weekend! Played with another library but it gets clogged up calling blocking functions and I’d like to avoid threads if possible.
Author
I’d love to hear out it works out for you. Good luck!
Don’t want to mess up your github with dumb questions… in the on_message callback, I occasionally need to call a long-running function (I’m placing a buy/sell limit order at best ask/bid and canceling/replacing it if price moves – to avoid paying maker fees and this can take a few seconds or even several minutes) and this stops me from getting ws messages until the function returns.
Is there a recommended way to asynchronously call my long-running function so I keep getting on_message callbacks?
Thanks.
Author
I would suggest making your long running function asynchronous and then use
loop.create_taskto add it to the loop where you need to inon_message. This will run your function in “parallel” with the rest of the websocket code.Great, I was just starting to try that!!
I know you didn’t sign up to debug my code, but…
I’ve tried:
loop = asyncio.get_event_loop()
task = loop.create_task(update_candles(parsed_time, num_candles=1, quote_curr=’ETH’, base_curr=’USD’))
also tried:
asyncio.ensure_future(update_candles(parsed_time, num_candles=1, quote_curr=’ETH’, base_curr=’USD’))
…
def async update_candles()
…
In both cases, it calls the update_candles() function but blocks
Author
update_candles will block once the loop calls it unless you’re able to
awaitwhatever is holding you up. Are you waiting for a state change? There are a few (more elegant) ways to do it, but I usually just set up a while loop that checks the condition and exits when the condition is met or otherwise callsawait asyncio.sleep(0)which releases control back to the event loop for a cycle before checking the condition again.update_candles() does this:
* makes buy/sell decision
* gets latest bid/ask (REST, but want to use ws ticker)
* places buy or sell at best bid or ask (REST)
* loops and gets order status
– If filled, then return
– if bid/ask changes then cancel existing order and replace at latest bid or ask
– else continue
There is no async code within update_candles.
Author
I’ve written similar code although I am using an asynchronous FIX client for placing/cancelling orders. I’m pulling the asks/bids and order statuses from the websocket feed. The bottleneck for me was watching for the bid/ask changes. That was the loop I put
await.asyncio.sleep(0)in.Oh, I see… ok, let me try to insert some await asyncio.sleep(0) in the loop
Awesome… adding a bunch of await asyncio.sleep(0) seems to do the trick and the main blocker now is just a synchronous REST call to get bid/ask… but I can rewrite that to get it from ws ticker and so things should be rocking now. Thanks so much. When I get this working, there will be invite to the private island it buys. 😉
And the happy ending is that slippage is down by like 75% using ws instead of rest. Thanks again.
Author
You are very welcome. I’m very happy it is working out for you. If you have any other questions or just want to talk shop, shoot me a message. Best of luck!
Author
On a side note, I have code for an asynchronous REST client for Coinbase Pro that I am cleaning up, writing tests and documentation for, etc. that I plan to add to CoPrA. No ETA on that unfortunately.
No worries… build one for Bitmex! afaik, they have no working Python library and don’t seem to care that much about.
Author
As of version 1.1.0, the CoPrA package has an asynchronous REST client. It only took 2 months, but it’s fully tested, documented, and live now.
Thanks for a thoughtful library!
Author
You are welcome! If you have comments, questions, or problems please let me know.
Awesome Library man. I’ve been playing with it here and there and I’ve noticed that sometimes Coinbase Pro will close with a code 1006 (Abnormal connection). Have you noticed this or known to occur before?
Author
I actually haven’t noticed that before. I’ve had some pretty long-running Websocket connections to Coinbase and only a few disconnects. They have never been frequent enough or persistent enough to motivate me to track them down. If you figure anything out, please feel free to share.
Yea, I’ve figured out I was not implementing on_close() properly as it has a reconnect function that fixes the 1006 disconnection issue. I don’t know why it gets disconnects or if any data is lost in between auto reconnects. I just recently realized my error so I haven’t had the time to look into it. I did realize that txid can be counted sequentially to find any holes. Anyways.. All in all pretty cool – its running long now. I will definitely track it to ground when I have time.
Oh Tony, I recall now there is a ping interval that could cause disconnects. Does the autobahn package have any pings it does to the websocket to keep it open? Here is a SO for the websockets library: https://stackoverflow.com/questions/54101923/1006-connection-closed-abnormally-error-with-python-3-7-websockets
See first answer.
Author
Marko, autobahn does provide for a configurable ping. I should probably document this better, but for now take a look at https://github.com/tpodlaski/copra/issues/6 and see if that helps you at all. As far as that goes, you can adjust any of the autobahn protocol options using client.setProtocolOptions. A good summary of the available options is here: https://autobahn.readthedocs.io/en/latest/websocket/programming.html#websocket-options. I hope this helps.
Hi, I’m passing an array called quotepair on to the client.
quotepair=[‘BTC-USDC’,’ETH-BTC’,’DAI-USDC’,’XLM-USD’,……………..
ws = Client(loop, Channel(‘ticker’, quotepair))
The plan is to get a stream of prices for each pair.
I am getting this stream.
{‘type’: ‘ticker’, ‘sequence’: 113942752, ‘product_id’: ‘XLM-BTC’, ‘price’: ‘0.00000719’, ‘open_24h’: ‘0.00000715’, ‘volume_24h’: ‘5148029.00000000’, ‘low_24h’: ‘0.00000700’, ‘high_24h’: ‘0.00000735’, ‘volume_30d’: ‘155312439.00000000’, ‘best_bid’: ‘0.00000719’, ‘best_ask’: ‘0.00000721’, ‘side’: ‘sell’, ‘time’: ‘2020-02-09T02:28:37.816461Z’, ‘trade_id’: 821112, ‘last_size’: ‘280’}
etc, etc, etc.
Which is good, but it streams onto the console. I’m trying to populate an array of [pair,prices]
How do I tame this stream?
I don’t want it streaming on to the console and I want to populate an array.
Thanks
Author
Hi Bob,
You need to subclass
copra.rest.Client.The
Clientclass has 4 callback methods that are called automatically during the client’s life cycle. The methodon_messageis called every time a new message is received. The default implementation simply prints the message to stdout.If you want to populate an array, subclass
copra.rest.Clientand overrideon_message. The ticker messages are the Python dictionaries you are seeing in the console, so inon_messageyou can inspect the message dictionary and do whatever you want with the data.You can read about the Client callback methods here: https://copra.readthedocs.io/en/latest/websocket/usage.html#callback-methods.
Hope this helps. Let me know if you have any more questions.
Yes, it helps immensely, was able to improve my polling time for all my pairs from 7.8 seconds to 0.001, that’s a 7800x improvement. Now I just have to figure out how to get the loaded price array into my algo.
Author
I am glad to hear you got it working. I found the best way to move data around ayncronously is with Asyncio Queues. Have the client’s on_message method put the data in the queue and build an ansync method connected to your algorithm that checks the queue for new data on each pass through the event loop .
Hey Tony,
Thanks for the library – it looks great. Just a quick question, is there a reason why you are using autobahn and not aiohttp for the websocket part? I am wondering about it since you do use aiohttp for the rest client.
Cheers!
Author
Glad you like the library! CoPrA started out as solely a websocket library, and for reasons I can no longer remember, I went with autobahn. When I decided later on to add REST, aiohttp was a no-brainer, but I haven’t had a reason or the motivation to redo the websocket parts (yet).
Tony – first of all – great job on your library. You indeed made it far simpler to engage the websocket interface for Coinbase. I got it working on a couple of different computers – doing exactly as you say – by subclassing the Copra Client and overriding on_message. I needed more compute power and bought the parts to assemble one – got it up and running etc. However, with the same code (python) that used to work on my laptop, on_message gets called successfully a number of times (for a few seconds) but then eventually hangs. Meaning no new messages. I am using OneDrive on Windows to write the market data to disk. But I do have exceptions handling on all of the file io (try blocks). I also override, on_close to see if I’m disconnecting but that doesn’t seem to be the case. I don’t expect you to be able to comment meaningfully on this – but in case you’ve seen something like this before thought I’d ask. Thanks again!