Some background:

WITCHCRAFT is an algorithmic trading system. It uses indicators to enter positions and momentum analysis to determine where to exit. There is no machine learning involved at any point in its analysis btw, but I intend to leverage ML to assist in tuning its profit efficiency down the line.

WITCHCRAFT trades Micro E-Minis futures (MES). It's important to understand what an MES is. Per Wikipedia, this is what a futures contract is:

In finance, a futures contract (sometimes called futures) is a standardized legal agreement to buy or sell something at a predetermined price at a specified time in the future, between parties not known to each other. The asset transacted is usually a commodity or financial instrument.

If you graph SPY's cost and superimpose it over the price of E-Mini futures, it's essentially a 1 to 1 fit. For all intents, both reflect the price disposition of the same collection of assets.

Here is the /MES candles compared to SPY (pink line) since 9:30 this morning:




So essentially if /MES increases $10 (10 points, we will get to this in a moment), the cost of SPY increases $1 in so many words.

But the magic here happens with the leverage.

Ok... "points"... We measure the rise/fall of /ES & /MES in "points". A $10 increase is referred to as a 10 point increase, etc etc.

In this scenario, we want to make $50 off SPY because we somehow know a 10 point positive shift in the S&P 500 index is about to hit.

So we need 50 shares of SPY, at an approximate cost of $17,000.

Ok, bet.

You know how much it costs to make $50 off a 10 point rise in the S&P 500 index?

$50. Seriously it costs $50 cash for one Micro E-Mini futures contract and it pays $50 if the S&P 500 index goes from 3380 to 3390.

UPDATE ok there was a bug where it didnt close out the order correctly so I closed out the order manually at $8.25 profit and fixed the bug. Moving on...

So those are Micro E-minis. And then we have regular E-Minis. Which cost $500, and also pay out 10x what Micro E-Minis do. So you would pay $500 for an E-mini contract in that scenario, and 10 points later it would have turned $500 profit.

This, people, is why every single truly sick degenerate market gambler loves E-minis. Nav Sarao loved E-minis. He went from trading one at a time to being worth $40 mil in a few years.

It has a lot in common with limit poker somehow but you get dealt infinite rivers and you can rake the pot or fold whenever you feel like it. Or something. I dont know.

The other side of this peculiarly sharp blade is that, like limit holdem, you stack up those losses fast. Every night they call in profits/losses. There are no pattern day trade rules for futures either. So most people scalp them constantly.

Search youtube for scalping e-minis and youll see what I mean. Most people will go for volume, looking to exit at 1 or 2 points. And that's fine but its worth noting that the rake on futures trades is pretty steep. Depending on your broker, you can pay $.80 to open and close a /MES future, and around $2.00 to open/close an /ES.

That shit adds up. It gets very hard to break even consistently when youre shelling out $400 a day in fees and believe me I've seen it happen.

So in the next post talk a little about how WITCHCRAFT deals with this, because although its technically a High Frequency Trading system, its wired to play swings, not scalps, which is pretty idiosyncratic compared to how most people do futures-fu.

So Im going to talk a little about how I'm approaching PnL and I'm being deliberately vague on a few points because thats part of my edge (hopefully), but I'm going to touch on some basic principles that lend themselves well to alg driven trading according to my math.

Short and sweet: I have extremely limited downsides, I enter statistically advantageous positions in terms of directional momentum, and I give my positions enough runway to rake significant profits with a scaling risk system that essentially locks in profit while diminishing the tolerance for reversals the deeper we get into profit. So when we end up in statistically anomalous profit, we basically get a real light trigger finger for reversals. But during smaller swells we tolerate larger reversals, but again try to make sure we exit green.

Another significant ah-hah moment was reading Jim Simons describe his 49/51 model, where he lost on 49% of his positions but raked larger amounts on his winning positions and won slightly more positions than he lost. For those who dont know, Jim Simons averages like 60%+ a year on his fund. It's on paper, you can look it up online.

At first I was allowing significant reversals and.. I mean.. it worked... but I suddenly realized that some of the positions I was staying in would develop over days (this is bad) and often take me into the red 20.. 30 points before they reversed enough to close green. And sometimes they didnt reverse before they tapped out. So there were algs where I would place 30 plays over 5 days, win 29 of them, but the loss on #30 was so egregious that it wiped my profits out.

Embracing the 49/41 Simons concept resolved that scenario.

On a more macro level, backtesting is showing this alg should be good for a comfortable, consistent minimum of 5 points a day / 25 points a week. With /MES thats about $125 profit a week. If I really do end up with that sort of profit at the end of the week (unlikely, lets be honest), I'll martingale up to 2 contracts per position, eventually switching over to /ES contracts because the fees are about 20% of what /MES contracts run once I put together enough cash to fade reversals without getting margin called.

Note that I said 'unlikely'. One of the fundamental truths to this is that algs inevitably perform better on back testing than they do on live accounts. There are a lot of reasons for this including spreads, slippage, orders not getting filled, yada yada it just all adds up. Hugely profitable algs fail when taken live, you just never know.

Some words on general architecture:

The hardest part of this, weirdly, was getting access to live streaming e-mini prices. If I knew then what I knew now, I would have opened an account with Interactive Brokers straight away, because they have the most well developed python friendly API on the market, by far. So that alone cost me a huge amount of effort chasing down one dead end after another.

I ended up committing to NinjaTrader because the fees got pretty low, its a super fast platform, and highly extensible. But again I cost myself a lot of blood/sweat/tear equity being unable to just solve my problems directly in python.

The brains of my system is a linux VM called MERLIN. MERLIN ingests live e-mini prices and feeds them into an indicator. All this is intraday, less than 10 minutes. MERLIN's indicator layer will return 3 values for neutral, upswing, downswing essentially. If neutral it simply sleeps until the next ingestion cycle. If a swing is clocked, it enters an appropriate position, watches behavior of the index, and then exits when it feels momentum is failing.

My first thought was simply running all this on the same host thats running NinjaTrader and I decided not to because I remember those backdoors in poker clients. I absolutely didnt want some overworked, underpaid tech downloading my source code as part of an 'investigation' into god knows what excuse they could come up with.

The solution was to host the crown jewels on another system and place calls to the Windows host (NT is running on Win10, named BERNE).

I'm mentioning this because getting commands executed on BERNE was surprisingly problematic. Most people come into this from a programming background, so they have a fairly robust understanding of C# and can pivot into message passing frameworks like zeroMQ with ease. Me, not so much.

This is an inelegant solution but it works:

I set up Apache on BERNE, and I wrote a bunch of python/pynput scripts that control mouse position/action. I have one for buy, one for sell, and one for close. Upswings buy a position then close to exit. Downswings sell a position then close to exit.

It works. MERLIN is able execute commands remotely on BERNE, BERNE cannot access the source code on MERLIN, and I did not have to spend thousands of dollars on Freelancer devs or dozens of hours of C# hacking to get it popping. That's a win in my book. A sloppy win to be sure but the game is the game.

Rewrote logging, generally trying to unfuck the stoploss mechanics during the 4:15 pause.

Also holy shit am I glad I didnt commit to a ToS based solution for this... jesus...

God damn sonatine, you really soaked up Python in just a few weeks eh?

I mean, Ive been using python for 25 years but to no real effect beyond scripting and delivering shellcode. And this isnt exactly object oriented, its a collection of Rube Goldberg-esque math triggers with some subprocess commands to do the wet work. I'll be earning my bones if I can port this all to a 100% python solution when I migrate to IB. And spoiler alert I'll probably tap in Fiverr devs whenever I get flustered. But the core logic is all written in python, yes.

So as a soft launch day I have nothing to complain about.

+ Enters positions accurately, consistently, profitably (so far).

- Logging/debugging hooks are absolute ass. Ive started to tear down and do properly but I'll have my hands full for a few days with that.

- Not exiting positions consistently and I need to go over notes and see whats happened between sandbox + live to make this an issue, but in retrospect I honestly think the problem is that I've been thinking in points during testing, and now I'm seeing the dollar amount.. so during testing I'm like ok cool I'm only down 2.5 points no biggie and now that I'm seeing that its actually down $12.50 I'm catching feelings, which is pretty fucking stupid but here we are.

- My broker got shitty with me tonight for fucking around with a paltry sum in my account which in fairness they warned me about already, so I had to wire in more money today. I'd love to end the day green tomorrow so I can start doing 2 contracts at a time instead of one so thats kind of an upside I suppose.

+ Alg trading still isnt illegal but fucking watch, I guarantee you some shits going to go down where they decide to levy a per-transaction tax on all stock/futures/options trades to cover the blooming deficit and somehow the big hedgecunts and bankercunts are going to be like ok no problem but how about retail traders cant HFT and suddenly thats going to be a law. Mark my fuckin words.

+ Hopefully before that fucks up my life I can sell this to a Chicago prop trading firm for a cool 5m headshot.

Also if my funds dont clear tomorrow I'm just going to papertrade online because honestly I should have just done that today.

it would appear that choppy 3 point swings are my motha fuckin kryptonite.

ok fuck it my current cash account balance is $648.85 and im literally going to let this bitch do whatever it wants for the rest of the night, im going to sleep.

got up early to check on this, down $20.50. looks like it got stop loss'd out of about 3 positions for about 3 points each but it took down one good swing for about 5.5 points. and what im seeing is that while having a 2 point stop-loss (that translates to 3.x points with slippage essentially) is a comfortable cushion in the day time, it might not be ideal for night time trades where we end up in so many of these choppy little ripples. tl;dr if i had implemented a less tolerant stop loss i would have woken up green today.

but then i end up potentially missing out on the big 20+ point swings that were essential to the PnL outcome in backtesting, so... tricky business, this.

ill leave the opening stop loss math alone for the time being.

something else occurred to me this morning; i need to dump all its position data into a db. im glad i got logging unfucked but having all this in a db is going to make things easier, period. ill use entry time as some sort of UID i guess...

interesting, while typing this the short position it was idling in caught about 1.5 points and now im only down around ... $12.50. tempting to flatten and taken profit but thats really not the point here is it.

the one tweak i did make is to encourage it take profit early on in its positions and be less tolerant of reversals once its green. with its current tick rate, it can enter/exit a 30 minute swing several times.. and given the horrific rake on these transactions id rather focus on modest gains than glory. the alternative is to increase the tick length which obviously exposes me to greater risk during the blackout period between updates. will crunch numbers and reconsider later this week or if i start to see consistent losses, whichever comes first.

so no sooner do i type about portnoy having covid and hit enter, the market plunges 25 points and after about 15 minutes i pulled the plug at -$72.50 for the day. ive made some modifications and restarted. odds are extremely good i switch back to papertrading at market close but im keeping my options open.