Memoirs – Becoming

Calling me a loyal son of Matar would be a stretch. If I was, I’d probably be in the Military, exercising my talents there, and furthering the goals of the Tribes.

Instead, I broke off, shortly after graduating; going my own way, forming my own corporation, and setting up business in Gallente space. I mean, I employ a bunch of ethnic Matar, sourced from all over New Eden, from former slaves, to those like myself, raised on a Gallente world.

There’s some guilt from that. They put a lot of time and effort into training me, and how do I repay them? I run off to do my own thing. That’s not exactly showing gratitude. But I just couldn’t hack the military life, far too many rules and regulations, people telling me what to do, and what to think.

I was raised by a family, proud of our Matari heritage, even if we didn’t live in our own Republic. No Clan, as such, just a Circle. I’m not going to go into exactly what we did. Skirting on the edge of legality at times. But they were good people. I don’t stay in contact as much as I should, but I helped them with repatriation. They didn’t want to go back, to be a drain on the Republic. I helped them get set up when they returned, transporting them myself, and arranging purchases on planet. They also help out with some of the slaves I’ve emancipated, getting them back into decent society, helping them with the side effects of their treatment.

What I wasn’t a fan of, at the time, was the fact they made me go to school. I mean, I learnt everything I needed for the family business from them, but they wanted a well-rounded education, and not to have “the Authorities” looking too closely at the clan, as to why I didn’t go. It’s not a stretch to say I did well. Top of my class, for the majority of the time. But I’m a capsuleer. I’m exceptional for a baseliner, but middle of the road, with my current peers.

I had some corporations come sniffing round, some pre-testing done, and some offers for scholarships made. I turned them down. Prideful, I wanted to help my people, so I travelled to the Republic, and tested there. Again, the offers came in, and I took one. That was a hard time. I almost quit on several occasions. University took work. Capsuleer training took willpower. Watching people ring out, walking away from everything they’d worked towards. But I hit that wall, and fought through it. Shame I couldn’t do the same, when it came to following orders, especially from a baseliner.

Egotistical? Yeah, sure. It’s hard not to be, when you’re part of that elite group that got through. Less than one percent of those who try, make it through to the end. And you need that ego, that feeling of self-importance, to carry you through the final test. Th unshakable knowledge that you will survive, even if the body you were born in doesn’t. I try not to let it out too much, but it’s always there.

You may have noticed I don’t have any tattoos. I used to. I shed them when I cut my ties with the Republic. My parents still argue with me about that. When I see them. I’m not the person I used to be. Those Tattoos weren’t mine any more. And how can I wear them, when I’ve abandoned my people? It just didn’t seem right.


Memoirs – Self Image

(It may not be good, but I don’t care. You don’t need to read it πŸ˜‰ )

Self Image

I’ve been asked before, why capsuleers aren’t all perfect specimens. Why some of us are scarred. Why some bear the legacy of the human endurance program. Why, when we can shape ourselves in pretty much any way we please, most of us chose to stay close to how we were, when we were baseliners.

I can’t speak for the others. One thing I’m not is a mind reader. And I guess I’m more unusual than most. I mean, every time I die, I wake up in a body which has had an eye and its arms replaced. There’s discomfort. There’s some relearning of how everything fits together, that I never had with a more natural body. [A snort of laughter] If you can call anything about these so-called clones natural.

But I think it’s all down to seeing a familiar face in the mirror when we wake up. I mean, we spend hours, days, sometimes weeks without being able to see ourselves. Where our bodies are the ships we pilot. Where our eyes are the drones we can send to see anything which we wish to. It’s a heady sensation, which can leave you feeling very small when you unplug. Limited. Vulnerable.

So, for me, I want to see myself in the mirror. Something to anchor myself, when I come down. Scars give you something to focus on. Something to say, “Yes, this is me.” Something to fool the mind into forgetting that this isn’t the same body as you were born in. Or even the one you woke up in that day. That might be half way across the cluster. Little tricks to ground you. Stop you forgetting who you are.

It’s one of the reasons I unplug when I can. Sure, there are risks to it. This me may die, because I’m not in my capsule, ready to be whisked away at a moment’s notice. But you can’t let that rule you. Even if this me may die, another can go on, branching from when I last got backed up. That won’t be me, but it’s a legacy, of sorts.

So that’s why we’re not all perfect. Because no-one wants to see a stranger in the mirror.

EULA Changes – Gambling

I’m not going to talk in detail about the EULA changes wrt the alpha clones. Which is the rest of them. There’s nothing substantial in there. Even the 90 days thing isn’t. (before, it was ‘account expired; we can delete it now’)

What I am going to do is talk about the other change. That to gambling with EVE Online currency and assets. This is a pretty big change. It won’t directly affect a lot of people, who just play EVE and wouldn’t think about giving their money to another player to gamble with. But it will affect people more than some will think. I’ll explain that a bit further down.


When I saw the devblog (Same time as the rest of you. The CSM didn’t get advance notice.), I was a little surprised. But not as much as I would have been a year ago. Over the last few months, there has been much more coverage in relation to gambling in video games, with some high-profile (in the industry) court cases. I was surprised, not because of the content, but only because of the timing. On reflection, with Alpha clones entering the scene, I shouldn’t have been surprised at that either.

CCP have a very big incentive to clean house. The removal of a subscription fee being required to play means that we’ll probably have a number of younger players joining us; younger players leading to greater scrutiny in this time of elevated concern about gambling in video games. CCP seriously don’t need that attention. It could prove very expensive for them, especially as it’s not something that they control.

While it’s unfortunate that some sites, such as EVE-Bet, are being hit by this, I fully understand why CCP are taking this tack. Unless you take the stance that all gambling sites are banned, you have to ban them one by one. White listing isn’t possible. Only black listing. So the ‘well behaved’ sites go down with the ones with a shadier reputation, instead of CCP opening their selves up to

Side Effects

This is where I have my only real problems with what is happening. Regardless of how you feel about them, some of the sites being forced to close did a lot of good for the EVE community, funding a variety of projects, and allowing for a richness of content which it will take time to recover. The common example is EVE-Bet. They’ve stayed above getting involved in the meta-game of EVE, instead funding a variety of streamers and other community content providers, such as EN24, Crossing Zebras and Eve-NT.

That’s something I’ll miss. I seriously doubt that donations will match that income. Even when you have something like Patreon, the number of people who will kick in real money is low. Probably because itΒ is real money. The number of people who will donate ISK is similar. I’m lucky. I don’t run up much in the way of EVE expenses. Some of the other sites, well, they’re going to have to seriously change how they do things.


Long story short, this is a positive change for the game. But not one without collateral damage.

With the introduction (a wee while ago) of the new eve market data endpoint, and the problems which surfaced at Eve Central while that was happening, I decided to create my own market data website. And so, I present

I posted it was out a few days ago, but thought I should write a little more about it, and the process of creating it.


At fanfest, in one of the round tables, I brought up (again) the idea of an endpoint to get the order book for an entire region at once. While most people just want aggregates, that’s not something that would be easy for CCP to provide for multiple items, at the same time. In part because the aggregation takes time, and because the multiple items makes caching that data a lot harder (different people asking for different lists of things). Lo and behold, a short time later, such an endpoint came into being. It’s paginated into batches of 30,000 items, which is enough for most regions, though The Forge has 9 pages.


So I rented a new server (with an offshoot of OVH called kimsufi) and set about building one. It’s a PHP site, running on an NGINX server, with a PostgreSQL database, and Redis for caching the aggregates.

The downloading script I wrote in python, and after an abortive attempts at aggregating the data, first in the database itself, then breaking it out into the script to process, I settled on using Pandas to do the aggregation. The script version would have taken over an hour to process. The Pandas version runs in a few minutes. This lets me do the grabbing of data once every 30 minutes, retaining it for a couple of days; that means you can look at snapshots of an order over that timeframe, to see how it changes.

That retention brought problems of its own. Not so much in keeping the data (each grab adds a couple of hundred meg of data and indexes.) but cleaning it up. Or being specific, the effects on the import, of having deleted it. Turns out the database doesn’t like it when you’re inserting and deleting 1.7 million rows every thirty minutes. It’s down to how it stores it. I won’t get into technical details, but it went from a couple of minutes, to over 15. which impacted kind of negatively on the performance of the site. The process wasn’t taking much CPU time, but it completely pegged the disk activity at 100%, and led to the site timing out. Not good.

How to solve this issue? One way would be to get a server with SSDs. Unfortunately, these are kind of expensive, relative to the one I’m using. I’m not made of money, after all. So I put together a Patreon campaign. If you want to contribute to the upkeep of the servers, I’d appreciate a pledge, however small it is. (Small is good in fact. My expenses aren’t that high. I’d feel bad about offloading everything to someone else)

However, a thought came to me recently. I’m using Postgres, and it can partition data, based on specific conditions. So I can have the data being stored in a partition based on the order set, and just truncate out partitions as they age out. This is far more efficient than deletion, and shouldn’t impact on the import speed. It’s not fully tested yet, but it’s looking somewhat better already. It’ll increase my data retention a bit (up to 200 samples (4 days ish), rather than the 96Β  (2 days) I was planning) but space isn’t too much of a concern that way. And partitions allow for more efficient queries.


I still need to write the API interface for it, so you can request an arbitrary number of aggregate/region combinations, but that isn’t too far out. And I’ll provide a sample method for importing into google. In addition, I’m providing the downloaded orderbook as a gzipped csv, for an undetermined period of time (it’s all dependant on space, tbh. and zipped they’re not huge)

I also need to decide how I’m going to winnow the aggregates. The API will only allow for ‘live’ data, as that’s all I’m keeping in redis, but all the data is also being kept in a database table. I’ll likely add a page so you can see how they change over time. To keep the table from becoming an absolute monster, I’m thinking I’ll keep only a week at 30 minute intervals, then averages for the day, if you want to see over a longer time frame.

In case you’re interested, I’m generating a weighted average (price* volume summed up, then divided by the total volume), the max, the min, and a weighted average of the lowest/highest 5% of the orders (lowest for sell, highest for buy) ignoring any price more than 100 times the lowest/highest price. It’ll help when someone puts in a buy order for a million Gilas, at 0.01 isk. Which otherwise completely screws with the average prices.


Oh, and I put together a very simple fleet tracker. Doesn’t do anything other than show you where fleet members are, and what they’re flying. Needs a CREST privilege, and a link you get from the fleet window. And for you to be fleet boss. I may expand it a little in the future. Maybe for invites too. Though that would require me to store the access and refresh token somewhere other than the user’s session.

As normal, all my code is on my github. Might be a little behind the release, but that’s purely down to me not committing it yet.

I’m open to suggestions, if you have them.

Anyway, Steve signing out.

New SDE Migration process

CCP Tellus is a wonderful person.

Until recently, the SDE (Static Data Extract) was a mix of YAML, an MS SQL server backup, and an SQLite Database for universe data. This was a bit of a pain to convert, with a whole bunch of steps.

Now, the SDE is being provided as a bunch of YAML files. Now, these, by themselves, aren’t a hugely useful format. But a single format makes it easier to write a conversion routine. I can convert into several formats, by running a single script (now that I’ve written it) This makes me a very happy man. is the repository for the new script. It’s written to convert into mysql, sqlite, postgres, and postgres in a specific schema. And it’s pretty simple to extend to other formats (MS SQL is a possibility, but as I can’t run that on my linux server, I’m not doing it at the moment)

In addition, with the SDE being in YAML, it’s a lot easier to generate the differences between versions. I’ve loaded it into a git repository. This is publicly available. git://

Feel free to look into it πŸ™‚


So, short version: I have a new SDE migration process. It may lead to slight differences in output, but the process is a lot easier for me. And if I get hit by a bus, it’s a lot easier for someone else to take over.

CSM: Tinfoil time with Uncle Steve

This is pure tinfoil, with no real substance.

Now, if I were a ruthless space dictator, lording it over many people, and I had a somewhat paranoid streak (as ruthless space dictators are want to have), I’d want to make sure that people voted the way I told them to. And have a way of catching spais who leak to my enemies.

The good news is, it’s possible with the current election system.

If I could send out, to my many sundries minions, a ballot that they should vote with, each tailored to that minion, I could, when all the ballots are released, tally each against a particular person.

There are 53 people standing for election. I only really care about, say, 11 of them. That leaves me 3 spaces I don’t care about, and 42 people I don’t really want to see elected.

So, each minion gets a ballot with those 11 people. The last three spaces are taken up my the other 42 people, in different combinations. so, that’s 42 x 41 x 40, or 68,880 different combinations. Enough for most space dictators.

I just send out all the ballots (bcced, of course, because I don’t want to share all the addresses they’ve entrusted me with. Of course.) and tally it up when it comes back. Or look up a particular ballot, when one is released anonymously. I just send out a message saying ‘oops, some of the ballots were mistakes.’, to allay any fears. πŸ˜‰

Tada, spai hunting, and voter control, all in one.

But this is pure tinfoil.

Using Crest with Excel: Crest Strikes Back

If you’ve read my older post on it, using Crest with Excel can be a complete pain in the ass. You have to load in a module, then write something in VBA, using it, because VBA doesn’t support json natively, and doing it with a user defined function, well, I never got that far, before throwing my hands up, and walking away.

Now, I got that working, but I didn’t like it. I ran into power query for a while, and it kinda worked, but I didn’t find a way to easily get it to pull a bunch of data, and reprocess it the way I wanted. So not so good for CREST prices.

So I was still on the lookout. And I recently came across xlwings, and it made me very happy indeed. I like Python, and I’m fairly good with it now. So, combining the two seemed like a winner.

After a little playing with it, I got it all working, and I now have an excel sheet, which does CREST lookups, and all it needs, is python, xlwings, and requests installed. So, to the instructions:

Install python. I’ve been using 2.7.10, but 3 should work just fine too. Make sure it’s in your path.

Install pywin32 (I’m using this one, for 2.7, 32 bit)

Using Pip, install xlwings. (you’ll do this in a command prompt)

[where you installed python, like c:\python27]\scripts\pip install xlwings

Using pip, install requests.

[where you installed python, like c:\python27]\scripts\pip install requests

using pip, install requests-security, to stop it bugging you

[where you installed python, like c:\python27]\scripts\pip install requests[security]

Now, go to a directory where you want to create your new workbook, using your command prompt. run the following command (myneatcrestbook, is the name of your new workbook. change it now if you want.):

xlwings quickstart myneatcrestbook

If this fails, it’s probably because your python scripts directory isn’t in your path. So try

[where you installed python, like c:\python27]\scripts\xlwings quickstart myneatcrestbook

With that done, you’ll find a new directory, containing an excel sheet, and a python file. open the excel sheet, and enable macros (it’ll probably just prompt you). You’ll also want to turn on the developers ribbon, in the file->options->customize ribbon option.

On the developers ribbon, click ‘macro security’, and check the ‘trust access to the VBA project object model’. hit ok.

Then click the ‘excel add-ins’ button, and click browse. Browse to [where python is]\Lib\site-packages\xlwings, and select the xlwings.xlam file. hit open, then check the box next to xlwings.

You should now have an xlwings ribbon entry.


That done, you can add code to the python file, come back to excel, and hit ‘import python UDFs’, to make them live.

When you want it to reprocess your functions (say, to update the data again, and get new prices) use ctrl+alt+F9

Below you’ll find a copy of a simple script to get the 5% average sell price. called with =crestSellPrices(34,10000002)

from xlwings import xlfunc

import requests

def crestSellPrices(typeid,regionid):
    for order in data['items']:
    # generate statistics
    if numberOfSellItems:
        while bought<fivePercent:
            if fivePercent > bought+sellPrice[fivePercentPrice]:
    return averageSellPrice

Solving the WarDec ‘problem’

I’ve made no effort to hide the fact that I think that WarDecs need work. That they’re inherently lopsided. That said, I don’t want to get rid of them. That would make highsec far too safe. People could anchor things with pretty much no chance of losing them. I’ve been kicking round a few ideas on how this could be ‘dealt’ with. That’s what I’m going to share here. I’ve mentioned it to CCP, but I don’t expect anything to happen soon, if it ever happened. They’ve got their hands full with some really large systems.

The core of the idea is simple:

Anchor a structure to declare a war. If that structure explodes, the war ends.

Not complicated, is it? Now a defender has a reason to undock and fight. If they win, the war is over. Maybe add an invulnerability to wardecs from that corp/alliance for a period (Week? Two weeks? Not long) Allies now have a concrete goal (Blow up the structure).

The key is it’s counterplay to the declaration itself. And there’s a degree of limiting for how many wardecs you run. Because every group you Dec can attack the structures which are running the wars for the other groups. Off time zones can be handled by Allies, if the Attacker tries to make it harder to attack, by setting it when no-one involved is on.

Yes, small groups will still be stomped by large groups. That’s always going to happen. But now there’s action they can take, rather than having to deal with an attacker that always fades if they manage to get a small fleet together.

There are a bunch of twiddly bits to set, of course.

  • How vulnerable are the structures? (I’d say at least one hour a day. Maybe a few extra hours which can be scattered around).
  • What do the structures cost? (Cheap)
  • What fuel do they structures take (This is the replacement for the isk cost to concord. Maybe some LP based things from pirate lp stores/drops, in addition to other fuel)
  • Where can it be anchored? (I’d suggest highsec, in the same constellation as the defenders head office)
  • When does the war end (I’d suggest the following downtime)
  • Edit: How is fuel burnt. (To keep some risk for the attacker, you put in fuel for 7 days. It’s all burnt to online it.)


And if you want a lore reason for the change?

Concord finally noticed a corrupt department taking bribes to turn off protection for people. They canned them. Now you have to hire hackers to do it for you. They have specialist equipment which requires microgravity, and can’t have a warp core in the same structure.

Summer Summit – Day 4

The final stretch. This has been a long week, and a draining one at that. It’s been pretty good, but if it went on longer, I’d probably end up breaking one of the windows to let some cold air into the room. (10 CSM, 2 Community Devs, + Devs from meetings. And the lights. it heats up rapidly)

  • EVE Producers
  • Team Security
  • FW & PVE
  • White Paper review
  • Community and Localization
  • Summit Retrospective

Eve Producers

Producers aren’t quite what you’d think. With the teams having a lot more control of what they do, their Managers exist to smooth the way for them. they take care of the regular HR stuff, while the dev teams get on with the regular stuff.

It was a good conversation, giving us insight into how things are managed.

Team Security


Just kidding πŸ˜‰ I’m expecting most of the minutes to be NDAd, but we had a good conversation with them, and I’m hopeful that you’ll see something from them soon.


The team isn’t combining these. They’re just the team working on both things. (There are a few things which are linked, but it’s not the reason they’re together). We talked abiut their plans (which should be, to some degree, in the minutes). It’s going to shake up the warzone, which is good. Stagnancy is bad.

White Paper review

We and CCP have been kicking round the white paper for a while, hoping to simplify it, and get a document which we all agree on. Not quite there yet, but really close.

Community and Localization

A session to talk general community stuff, with the community team, and the people handling localization. Covered a bunch of things.

Summit Retrospective

An After Action Report on the summit itself, and how we thought it went, and how it could have gone better. Really worthwhile, though not, tbh, hugely interesting for people who weren’t there. Lots went well. A few things, not so much. (We were in their biggest meeting room. But with all of us in there, and it not having good cooling, we boiled. Next time, probably a smaller one, but with an exterior door)



And so the summit ends. It went better than I expected, though that’s down to bad expectations on my side. I should remember that face to face nearly always goes better. When you deal in mostly text, or just audio, a lot is lost. When we could see, and compensate, things went smoothly.

Summer Summit – Day 3


So hungry. I may need to eat one of the other CSM members. It’s been a fairly long, but productive and interesting day.

  • QA team session
  • EVE Metrics / In-game economy
  • Launcher / download on demand
  • Structures
  • EVE Store
  • Art session
  • UI session
  • Fanfest

QA Team Session

The day started with a huge Scotsman, with a large beard talking to us. As you might expect, this immediately made me feel at home. We talked a bit about how the QA team do their work (not the actual QA bit. The infrastructure around it) and where they think they’re strong, and where they’re not so good. Saw some graphs, heard some suggestions. A good meeting.

EVE Metrics / In-game economy

Heard some good news in this session, but it’s not my place to announce things. So I’m not going to. But again, I’m entirely happy with how this session went. And seeing some metrics generated on the fly (from the underlying logs) is always interesting.

Launcher / download on demand

omgomgomg. I will say no more.


We talked about structures. Both the Citadels, and the future structures which are coming later. We mostly focused on places where we weren’t entirely sure about things. I don’t want to talk about this too much, as the NDA applies at the moment. But I’m hopeful that there’ll be more in the minutes.

EVE Store

I want to buy stuff! And I want it to not cost me an arm and a leg! And CCP appears to have listened to this. As with things involving partnerships with other companies, CCP are going to hold this close to the vest until it’s actually ready. With luck, that’ll not be too much longer.

I asked about the card game, and the possibility of print on demand with a partner. As this was kind of a blind side question, I didn’t get a firm response. I didn’t expect one, just wanted to plant the idea.

Art session

We were shown a bunch of art stuff. It was cool. It’s not particularly possible to talk about, without having pictures, so I’m not going to. But Eve has yet more prettiness coming. I asked about hats (and other accessories). I got some dirty looks. As per last time, a lot of it is having to deal with things like hair as well, which is just difficult to work with, as customizable as it is. To make sure it doesn’t clip, or just look weird. It’s a customization thing. People have their portraits, and some people take a lot of effort over them. Some accessories naturally fit, like hats, which would sit inside the portrait.

UI session

CCP has a new team devoted to the User Interface and User Experience. (rather than embedded people, in other teams). They showed us some concept stuff they’ve been kicking round, we brought up some pain points on the current UI. A good productive session, but again, one that, without the pictures, isn’t simple or interesting to talk about.


We talked about Fanfest, with the person who organised it last year, about where we thought it went well, and where it could have done better. An After Action Report, so to speak. We had a bunch of comments, they had questions.


Today felt incredibly productive, covering a lot of ground, and having a lot of discussions. I’m not sure how much of that will come across in the minutes (They’re always a cut down version, unless you have a stenographer, and are willing to wade through reams of text), but it went well today.