Tag Archives: development

Understanding the Eve Online SDE – Part 1

How many parts is something I can’t yet answer. As many as it takes.

The SDE is a Static Data Extract from Eve Online. It contains information like where stars and planets are, how much Tritanium goes into a Typhoon, what skills are needed for which modules and so on. It’s a useful bag of information if you’re going to be doing any development of third party tools for the game. Much (but not all) of the information is also available directly from ESI, where it’s potentially more up to data. However, it can be a lot more time consuming to process there, and some is just not available. Like what blueprints need.

The official SDE direct from CCP

I’m not going to go into a great deal of depth on this. It’s not what most people use.

From https://developers.eveonline.com/resource/resources you can download the file sde-TRANQUILITY.zip

This is the main source for the SDE, and what I use to create my own versions. The main reason most people don’t use is, is that it’s a fuckton (metric, not imperial) of yaml files. Great for passing data around, not so good for looking up something.

CCP tend to update this a wee while after a release happens. Sometimes they need poked to do it. If you’re going to work directly from it, be aware that not all yaml parsers are created equal and can handle the spec a little differently. pyYAML is probably the one to use if you can, with libyaml to stop it taking forever.

The SDE release from yours truly

For a little history, CCP used to release the SDE as a MS SQL server dump. When I got started, someone else did the conversions to mysql which I used. Then they got slower and slower, and I got impatient, finally working out how to do it myself. A few years later, CCP started converting how they did their data versioning internally, moving from what they called BSD to FSD. Which meant they moved to yaml files for much of it. At this point, I totally rewrote the converter process I was using. And because I could, I set it to target a number of different database. If I get hit by a bus, you can find it at https://github.com/fuzzysteve/yamlloader It’s python 2.7, but should be simple enough to convert to 3 if you want to.

https://www.fuzzwork.co.uk/dump/ is where I keep the conversions. I don’t promise to keep them for all time, but older versions are available if you want them. the latest version is always in the latest folder or linked to with the latest links in the root.

If you’re wondering about the structure and names, most of that is a hold over from the original SDE, so that applications needed minimal changes.

Available are:

  • MySQL, both a complete dump, and individual tables
  • SQLite, a complete database file.
  • Postgresql, pg_dump file for use with pg_restore
    • the normal one, which puts all the tables into the public schema
    • the ‘schema’ one, which puts all the tables into the evesde schema. I recommend this one for people starting out with postgres. Easier to update.
  • MS SQL server, a Data Tier Application export.
  • CSV files, now in gzip, bzip2, and plain format.

The Tables (In my conversion)

There are many tables (or CSV files) in the Eve SDE. However, these tend to fall into a small number of groups only a few of which are wanted.

The ‘Stuff’ tables

invTypes

This is the core table for many applications. If you have something which isn’t a unique item, then this is probably where you’ll want to look it up. Have a typeID from ESI which you want to look up? This is that table. Skills, ships, trade goods, generic planet types, all that good stuff. There’s a published column here which basically says whether it’s actively in the game or not.

invGroups and invCategories

invTypes has a groupID column. That links to invGroups, which has a categoryID which links to invCategories. invCategories contains a list of the types of stuff which exist. Like Ships, Skills, Planets, Modules, and so on. invGroups breaks that down further to things like Assault Frigates, Battleships, Science Skills, Asteroids and so on.

Example: you want all the ships in the game. you look up invCategories and find that ships are categoryID 6 (rather than always joining in the invCategories table.) 

select typeName from invTypes join invGroups on invTypes.groupID=invGroups.groupID and invGroups.categoryID=6;

invMarketGroups

Much like invGroups, invMarketGroups is referred to by marketGroupID in invTypes. a little complexity is added, because each market group can have a parentMarketGroupID. This is so you can have each assault frigate in a market group for their race, that market group in the assault frigates group, that market group in advanced frigates, that in Frigates, and that in Ships. (ships doesn’t have a parent). Getting the full tree in pure SQL is a complete pain in the ass.

invMetaGroups and invMetaTypes

invMetaGroups defines the various meta level groups. Like tech 1, tech 2, faction and so on. invMetaTypes links that back to invTypes so you can list all the T2 ships by joining it in. It also has the parentTypeID in case you want to see what the T1 version is.

select typeName from invTypes join invGroups on invTypes.groupID=invGroups.groupID and invGroups.categoryID=6 join invMetaTypes on invTypes.typeID=invMetaTypes.typeID and invMetaTypes.metaGroupID=2

invTraits

All those handy ship bonuses have to live somewhere. Links to invTypes for the ship type (typeID) and to invTypes for the skill (skillID). -1 is a bonus which isn’t related to a skill. unitID links to eveUnits so you know if it’s a percentage or what.

invFlags

This is related to the asset table in Eve. If you get back a list of your assets, they’ll have flags on them. You won’t see hangars and so on. The Flags determine where they are. Something with flag 11 is in the first lowslot on a ship, for example. This is why shared hangars are painful to implement.

invTypeMaterials

This _used_ to be used for making things. It’s not any more. it’s just what you get out of something when you recycle it. typeID links to invTypes for what you’re recycling, materialTypeID links to invTypes and is what you get out. quantity is what you’d get on a perfect (100%) refine.

invVolumes

When things change size as they’re packaged, the volume in invTypes becomes less useful. This is the volume of things which are packaged (when it’s different)

invItems

A table of unique items, where they are, who owns them and what they are. I don’t think I’ve ever seen anyone but CCP use this.

invNames and invUniqueNames

the names for the things in invItems. Don’t know what kind of something a thing is, but have an itemID? It might be in there. There’s almost always a better table to look at to get this information, because you normally know.

invPositions

Where the things in invItems are in space. like with invNames, it’s unlikely you’ll want to use this.

Industry Tables

When Crius came out, CCP changed up everything. And had a new yaml data file just for making shit. So I created a number of totally new tables from it. If you don’t like the format, it’s all my fault. Try and do better. I dare you.

A common column in these tables is the activityID column. This is to split them up into the various things you can do with a blueprint. Like manufacturing, researching TE, copying, reacting, inventing and so on. Yes, reactions are just blueprints like anything else now.

1Manufacturing
3Researching Time Efficiency
4Researching Material Efficiency
5Copying
8Invention
11Reactions
The Activity IDs

industryActivity

typeID is the blueprint typeid in invTypes. time is how long it takes to do that activity before modifiers.

industryActivityMaterials

Which materialTypeIDs from invTypes are used for an activityID when using a typeID (blueprint) from invTypes and the quantity before modifiers

industryActivityProducts

What quantity of productTypeIDs from invTypes are produced per run of a typeID (blueprint) from invTypes for a specific activityID

industryActivitySkills

What skillID from invTypes is required to do activityID on a typeID (blueprint) from invTypes and what level it needs to be.

industryActivityProbabilities

Invention has different probability depending on the typeID of the blueprint (invTypes) with a handy productTypeID to work out what it makes.

industryBlueprints

maxProductionLimit is how many blueprint copies can be on a BPC.

Planetary Interaction.

planetSchematics

The various things you can make on a planet and how long they take.

planetSchematicsPinMap

Just links the schematics to the kind of structure on planet which makes them. Like basic industry. though it is by planet type as well.

planetSchematicsTypeMap

Takes the schematicID from planetSchematics, and tells you the quantity of what typeIDs you need to put in (isInput=1) to get typeID out (isInput=0)

The Dogma Tables

Dogma is Eve’s engine for knowing stuff about things. What slot does a module go into? Dogma. What skills are required for something? Dogma. How much damage does a gun do? Dogma (though it’s hard to work out. yay)

dgmAttributeTypes

The table with the description of the various dogma Attributes. by attributeID

dgmTypeAttributes

The table which links dogma attributes attributeID to item types invTypes typeid. These days, just use valueFloat for the value. everything is condensed into that now. Handy for things like how many low slots something has. Or which skills are required. (with a separate attribute for which level it’s needed at. yay)

To see all the attributes and what they do, for a Rifter (typeid 587) try:

select * from dgmTypeAttributes join dgmAttributeTypes on dgmTypeAttributes.attributeID=dgmAttributeTypes.attributeID where typeID=587;

dgmEffects

The various bits of dogma which are just flags on items, rather than things with values. like if it’s a low slot module or not. This describes those flags.

dgmTypeEffects

Linking typeID from invTypes to effectID in dgmEffects

dgmAttributeCategories

Just splits the various attributes up into categories. Like if it’s a fitting attribute, shield, or so on. categoryID in dgmAttributeTypes

dgmExpressions

This is stuff which relates to how things actually work in eve. the code behind it. I don’t understand it well enough to explain it.

Map Tables

mapDenormalize

This has pretty much every celestial in it, and where it is, in meters, in its star system, with the star at 0,0,0

Links to invTypes so you know what some is (planet etc) by typeID and invGroups by groupID. links to mapSolarSystems with solarSystemID, mapConstellations with constellationID mapRegions with regionID.

What might not be obvious is wormhole effects are here, with a groupid of secondary sun.

mapJumps

Less useful than it might sound, it tells you which pairs of stargates link up. You probably want mapSolarSystemJumps

mapSolarSystemJumps

what solarSystemIDs have stargates to what other solarSystemIDs. also has regions and constellations

mapSolarSystems

So you can find out information about solarsystems by solarSystemID. Names, regionIDs, ConstellationIDs, where they are in the universe in meters. (yes, light year ranges stored in meters) and their security rating.

mapRegions

Simiar information, but for Regions.

mapConstellations

Take a wild guess 😉

The end, for now

There are still a bunch of tables which I’ve not talked about yet. Most are pretty self explanatory, but I’ll get to them another time. This should be enough to get you going.

market.fuzzwork.co.uk

2 Replies

With the introduction (a wee while ago) of the new eve market data endpoint, and the problems which surfaced at Eve Central while that was happening, I decided to create my own market data website. And so, I present market.fuzzwork.co.uk

I posted it was out a few days ago, but thought I should write a little more about it, and the process of creating it.

Background

At fanfest, in one of the round tables, I brought up (again) the idea of an endpoint to get the order book for an entire region at once. While most people just want aggregates, that’s not something that would be easy for CCP to provide for multiple items, at the same time. In part because the aggregation takes time, and because the multiple items makes caching that data a lot harder (different people asking for different lists of things). Lo and behold, a short time later, such an endpoint came into being. It’s paginated into batches of 30,000 items, which is enough for most regions, though The Forge has 9 pages.

Details

So I rented a new server (with an offshoot of OVH called kimsufi) and set about building one. It’s a PHP site, running on an NGINX server, with a PostgreSQL database, and Redis for caching the aggregates.

The downloading script I wrote in python, and after an abortive attempts at aggregating the data, first in the database itself, then breaking it out into the script to process, I settled on using Pandas to do the aggregation. The script version would have taken over an hour to process. The Pandas version runs in a few minutes. This lets me do the grabbing of data once every 30 minutes, retaining it for a couple of days; that means you can look at snapshots of an order over that timeframe, to see how it changes.

That retention brought problems of its own. Not so much in keeping the data (each grab adds a couple of hundred meg of data and indexes.) but cleaning it up. Or being specific, the effects on the import, of having deleted it. Turns out the database doesn’t like it when you’re inserting and deleting 1.7 million rows every thirty minutes. It’s down to how it stores it. I won’t get into technical details, but it went from a couple of minutes, to over 15. which impacted kind of negatively on the performance of the site. The process wasn’t taking much CPU time, but it completely pegged the disk activity at 100%, and led to the site timing out. Not good.

How to solve this issue? One way would be to get a server with SSDs. Unfortunately, these are kind of expensive, relative to the one I’m using. I’m not made of money, after all. So I put together a Patreon campaign. If you want to contribute to the upkeep of the servers, I’d appreciate a pledge, however small it is. (Small is good in fact. My expenses aren’t that high. I’d feel bad about offloading everything to someone else)

However, a thought came to me recently. I’m using Postgres, and it can partition data, based on specific conditions. So I can have the data being stored in a partition based on the order set, and just truncate out partitions as they age out. This is far more efficient than deletion, and shouldn’t impact on the import speed. It’s not fully tested yet, but it’s looking somewhat better already. It’ll increase my data retention a bit (up to 200 samples (4 days ish), rather than the 96  (2 days) I was planning) but space isn’t too much of a concern that way. And partitions allow for more efficient queries.

Future

I still need to write the API interface for it, so you can request an arbitrary number of aggregate/region combinations, but that isn’t too far out. And I’ll provide a sample method for importing into google. In addition, I’m providing the downloaded orderbook as a gzipped csv, for an undetermined period of time (it’s all dependant on space, tbh. and zipped they’re not huge)

I also need to decide how I’m going to winnow the aggregates. The API will only allow for ‘live’ data, as that’s all I’m keeping in redis, but all the data is also being kept in a database table. I’ll likely add a page so you can see how they change over time. To keep the table from becoming an absolute monster, I’m thinking I’ll keep only a week at 30 minute intervals, then averages for the day, if you want to see over a longer time frame.

In case you’re interested, I’m generating a weighted average (price* volume summed up, then divided by the total volume), the max, the min, and a weighted average of the lowest/highest 5% of the orders (lowest for sell, highest for buy) ignoring any price more than 100 times the lowest/highest price. It’ll help when someone puts in a buy order for a million Gilas, at 0.01 isk. Which otherwise completely screws with the average prices.

Other

Oh, and I put together a very simple fleet tracker. Doesn’t do anything other than show you where fleet members are, and what they’re flying. Needs a CREST privilege, and a link you get from the fleet window. And for you to be fleet boss. I may expand it a little in the future. Maybe for invites too. Though that would require me to store the access and refresh token somewhere other than the user’s session.

As normal, all my code is on my github. Might be a little behind the release, but that’s purely down to me not committing it yet.

I’m open to suggestions, if you have them.

Anyway, Steve signing out.

BP Calculator: T2 Invention update

Leave a reply

Just another quick update.

The Form Entry page now has another submit button, called ‘Enter list – T2 Invention’. If you use this, it will take a list of T1 blueprints which can be invented from (ignoring anything else) and produce a list of everything you can make, along with a basic price for them. It doesn’t deal with decryptors, so it’s not doing all the work for you. And I’m not going to simplify it to just take a list of blueprints without the rest of the line.

But if you already have a list of blueprints, it’ll help pick out the ones you might want to look at. Just remember, there’s no market volume data. Just price data.

Updates and future plans

1 Reply

If you were to drop by the blueprint calculator, you’ll have noticed a few changes.

New features

  • Regional prices. You can now pick the region on the price list. It will always  default to Jita for now.
  • Collapsible price list. It will remember the  state you left it in.
  • Parser for a blueprint list. If you open up the S&I interface on the blueprint list (or corp blueprint), then double click on the location you’ll get a window you cut and paste from. Paste it in, that you’ll get a list of links, with a profit per unit.
  • General optimization. Performance since I added the table sorting has suffered. It should now be back up to normal.
  • Added a while back: Decryptors in the invention figures.

Future plans

  • Once the SSO becomes available, you’ll be able to log in, and store blueprints, if you want to. This will be on the server, rather than in the client. I’ll probably leave the option to store with cookies too, if you don’t trust me with that. Other preferences will be stored
  • Once CREST becomes available, with a character sheet end point, your skills (if desired) will autofill.
  • Inclusion of invention prices on the parsed results.
  • Inclusion of isk/hr (ish) on the parsed results
  • Addition of a menu bar, to bounce between applications.

 

 

SSO/CREST:

Don’t worry about the security of SSO. The way OAUTH2 works is:

  • You hit the login button, which sends you to login.eveonline.com with a token that identifies where you’re coming from.
  • You log into Eve online, and pick the character you want to be.
  • You get sent back to the source site, with a long token.
  • The source site sends the token to eve online asking ‘who is this?’
  • Eve responds, identifying who you are and invalidating the token for future use. (CREST works a little differently but not significantly.)

At no time do you give anyone except CCP your password. Any third party site asking for it is either badly coded, or a scam. Ideally you’d log into the forums, and then you won’t need to type in a username or password into any site.

Using the Eve Central API with PHP

3 Replies

We’re going back to basics here. One thing that many sites, which are using the SDE from Eve, want to do, is access price data from EVE Central api. I don’t do this myself, having a EMDR relay, and consumer to provide my market data, but back when I was starting out, I did this. It’s a pretty simple thing to do, the only pain in the neck being the data structures that PHP wants to use, when you’re using XML. It likes throwing in arrays where you don’t think it should, which can be a bit of a pain. If you ever need to debug this, var_dump() will be your friend, showing you exactly how the data has been structured.

So, here’s a fairly basic bit of code. It asks eve central for the information for Tritanium, then prints out the so called ‘Percentile’ price. This is the price if someone was to buy 5% of the market, then average the cost out. It’s handy for ignoring the outliers, if not entirely accurate. The code is more complicated than it needs to be, but we’ll get to why in the second example.

$typeid=34;
$url="http://api.eve-central.com/api/marketstat?regionlimit=10000002&typeid=".$typeid;
$pricexml=file_get_contents($url);
$xml=new SimpleXMLElement($pricexml);
$item=$xml->xpath('/evec_api/marketstat/type[@id='.$typeid.']');
$price= (float) $item[0]->sell->percentile;
$price=round($price,2);
echo $price;

Fairly simple, really. You get the type id for Tritanium, the url that will pull back the data from the forge, mash them together, then use file_get_contents to retrieve them. Please not, some PHP installs will stop this from working. There’s a more complicated version at the bottom which may get you round this; functionally identical but using curl instead of file_get_contents.
Once you have the output from EVE Central, you push it into the SimpleXML parser, grab the bit that’s just about tritanium (the XPath bit does this. This is the over complicated bit), then pull just the sell percentile out of it. The reason for the [0] is because the Xpath bit could have returned more than one entry. In this case it can’t, but the parser is taking the safe option and giving you an array. The (float) is there as without it, you’re dealing with a SimpleXML data type, which is a pain to do math with. (float) just converts it into a regular number.

Now for the more complicated version. Eve central allows for multiple items to be queried at the same time. This is why I put the xpath in there. It allows us to target a specific type being returned, rather than having to go through each in turn to see if it’s the one we want, before continuing.

    $typeids=array(34,35);
    $url="http://api.eve-central.com/api/marketstat?regionlimit=10000002&typeid=".join('&typeid=',$typeids);
    $pricexml=file_get_contents($url);
    $xml=new SimpleXMLElement($pricexml);
    foreach($typeids as $typeid)
    {
        $item=$xml->xpath('/evec_api/marketstat/type[@id='.$typeid.']');
        $price= (float) $item[0]->sell->percentile;
        $price=round($price,2);
        echo $typeid." ".$price."\n";
    }

This time, instead of a single type id, we’re asking for multiple. We store them in an array, then mash that array into the url, with a join (or implode) which adds the right number of &typeid= to it. Then it goes on as before until just after the xml is parsed, where it goes and grabs each individual element out in turn, iterating through the array.

If you wanted to pull the minimum buy price, you’d replace the $price= (float) $item[0]->sell->percentile; with $price= (float) $item[0]->buy->min; and so on. It’s fairly simple.

Now, the curl version, of the first bit of code:

$typeid=34;
$url="http://api.eve-central.com/api/marketstat?regionlimit=10000002&typeid=".$typeid;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$response = curl_exec($ch);
if($response === false)
{
    echo 'Curl error: ' . curl_error($ch);
}
else
{
    $xml=new SimpleXMLElement($response);
    $item=$xml->xpath('/evec_api/marketstat/type[@id='.$typeid.']');
    $price= (float) $item[0]->sell->percentile;
    $price=round($price,2);
    echo $price;
}

It’s a bit more complicated, but it gives you a way of dealing with a remote site not responding. Perhaps looking at a different site for price data, for example.

Remember, the more calls you have on remote sites, the slower your pages will be to load. Consider caching the price data locally, perhaps in a database, or if you have access to it, memcache (My preference. but it’s uncommon)

Dev Track Day 2 – Part 2

Leave a reply

This was the WebGL session.

It’s great. Really really great.

The only problem is, potentially, delivering the models to the clients. Other than that, it’s wonderful. If you can take the loading time, it’s great.

https://github.com/ccpgames/ go fork it.

 

Not a huge amount to say really. It was demos of graphics stuff. which looked great.

Dev Track – Day 2 – Part 1

Leave a reply

Just a small gap between the SSO session and the WebGL session.

The SSO looks to be fairly simple to work with. It’s a shame that the demo mobile applications which were demonstrated didn’t use a native browser, using an embedded one, which is far from good practice with OAUTH2, but they do make them easier to understand, with less explanation needed (Custom protocols for the call back to get the token to the application, for example. I believe Aura does this with getting the keys from the site.).

It was fairly obvious, from some of the questions being asked, that there were some fundamental misunderstandings, but in the end, it works well enough. More demonstration code will be needed, but I’m sure the community will provide some. I’ll probably do it, when I kick it around myself.

There was some discussion at the end, about situations like character transfers not being obvious to third party sites, but the devs seem receptive to possibly adding a userid characterid based hash. So you can’t tie users together, but you can tell when one is transferred.

No ETA on deployment, unfortunately. As it’s tied to the dev license, I wouldn’t expect it until at least the next draft (30 days+)

There was also a request from the devs to make sure that we make it obvious on the forums if we’d like an updated IGB, and two factor authentication. I’ve added posts in the tech lab, so if you could go and wave in them, I’d appreciate it.

Useful bits and pieces – Eve SDE

4 Replies

Once you get comfortable with the CCP way of doing things, the SDE isn’t too bad to work with. You’ll have to dig around in it to get some of the numbers you want, and sometimes it’s the opposite way round from what you’d expect, but it’s not too bad, most of the time. Here are a few more useful snippets for getting information on things.
Continue reading →

Eve SDE SQL – Blueprint Details – Part 2

3 Replies

I covered the basics of Blueprint details from the SDE back in my last post, which covers most of what people actually want to know. There are a few extra bits which the more dedicated coders will want. They’re simple enough to work out, but it’s time you might not want to have to look at. Consider this a round up post.

Perfect Blueprints

Perfecting blueprints is, in many cases, a waste of time. You want ‘good enough’, rather than perfect. It’s also unattainable for some. A Raven would take you over 4 million days to perfect. And half of that time (or so) would be to save a single Trit.

I don’t have a formula for a ‘good enough’ level. It’s very dependent on the types of materials being used. The Formula for perfect, for each material, is:

floor(Quantity*(\frac{(\frac{Waste Factor}{100})}{0.5})

If you want the perfect for an entire blueprint, pick the material with the highest quantity.

As before, the Waste Factor comes out of invBlueprintTypes.

Production Times

More details from invTypeBlueprints.

If the PE of the blueprint is 0 or above, the total time in seconds is:

productionTime*(1-(\frac{productivityModifier}{productionTime}*\frac{Blueprint PE}{1+Blueprint PE}))

If the PE of the blueprint below 0, the total time in seconds is:
productionTime*(1-(\frac{productivityModifier}{productionTime}*(Blueprint PE-1))

To then adjust it for your industry level

Time from Above * (1 - (0.04 * Industry))

So to recap, the time in the database is the maximum time it’ll take to manufacture something (assuming it’s not a negative PE), with everything else reducing the time. The changes also fall off rapidly. PE beyond 5 or so is often worthless.

Materials to invent off of something

select t.typeid,t.typeName, r.quantity
from ramTypeRequirements r,invTypes t,invBlueprintTypes bt 
where r.requiredTypeID = t.typeID 
and r.typeID = bt.blueprintTypeID
and r.activityID = 8
and bt.productTypeID=?;

The bind variable is what you’re making. You could simplify the query if you had the blueprint ID already, eliminating the join to invBlueprintTypes, going straight to ramTypeRequirements. It’s possible to pull the damage too, but the SDE currently says the Interface is used up.

If you want to eliminate the interface from the pulled materials, use this sql. It’s eliminating interface, by ignoring the group it belongs to. To pull just Interface, then turn the != into a =

select invTypes.typeid,invTypes.typename,ramTypeRequirements.quantity
from ramTypeRequirements,invBlueprintTypes,invTypes 
where producttypeid=?
and ramTypeRequirements.typeid=invBlueprintTypes.blueprintTypeID
and activityid=8
and invTypes.typeid=requiredTypeID
and groupid !=716;

Skills to manufacture something

select t.typeName Name, r.quantity Quantity
from ramTypeRequirements r,invTypes t,invBlueprintTypes bt,invGroups g
where r.requiredTypeID = t.typeID
and r.typeID = bt.blueprintTypeID
and r.activityID = 1
and bt.productTypeID=?
and g.categoryID = 16
and t.groupID = g.groupID

The bind variable is what you’re making. You could simplify the query if you had the blueprint ID already, eliminating the join to invBlueprintTypes, going straight to ramTypeRequirements.

Skills to invent something

This is a little bit of a pain. The skills are tied to the datacores and interface, rather than the job. Specifically they’re in dgmTypeAttributes, under the attributeIDs of 182 for the skill, and 277 for the level of the skill. As they’re done that way, pulling it with a single sql query is somewhat difficult, with a pivot table required (use if conditions, along with a max, then group them). When I’ve done it, I’ve just used 2 queries, and handled the pivoting in the code.

The skills

select t.typeid,t.typeName skill,r.requiredTypeID
from ramTypeRequirements r,invTypes t,invBlueprintTypes bt,
dgmTypeAttributes dta 
where coalesce(dta.valueInt,dta.valueFloat)= t.typeID
and r.typeID = bt.blueprintTypeID
and r.activityID = 8
and bt.productTypeID=?
and dta.typeid=r.requiredTypeID
and attributeID=182

The requiredTypeID refers to material you need the skill for. In this case, datacores and interfaces. Everything else is the skill.
The bind is what you’re making.

The skill levels

select r.requiredTypeID,coalesce(dta.valueInt,dta.valueFloat) level
from ramTypeRequirements r,invBlueprintTypes bt,dgmTypeAttributes dta
where r.typeID = bt.blueprintTypeID
and r.activityID = 8
and bt.productTypeID=587
and dta.typeid=r.requiredTypeID
and attributeID=277

The requiredTypeID refers to material you need the skill for. Level is the minimum level for the skill.

In the previous 2 queries you’ll have seen the use of coalesce. That’s because CCP seem to have had trouble deciding which to store the value in. It’s normally the valueInt, but not always.

Research times

researchMaterialTime and researchProductivityTime are straight out of invBlueprintTypes.

Material Efficency research

Per level:
researchMaterialTime*(1-(Your Metallurgy*0.05))

Productivity Level research

Per level:
researchProductivityTime *(1-(Your Research*0.05))

Times are all in seconds.

Getting Basic Blueprint Information

select productionTime,wasteFactor,productivityModifier,researchProductivityTime,researchMaterialTime
from invBlueprintTypes
where productTypeID=?;

This is most of the information needed in previous equations.

Is this obtainable through invention?

select parentTypeID
from invMetaTypes
where typeid=?
and metaGroupID=2

This will only return a result if you can invent the blueprint from another one. The parentTypeID is the typeid of the original type. Such as a rifter, if you’re looking at a wolf.

Chance to invent

This isn’t actually in the SDE. However, I found some handy sql to stick it into an table you can use for lookups. The type id is that of the product, rather than the blueprint.

create table inventionChance(typeid int,chance float);

insert into inventionChance (typeid,chance)
select typeid,CASE
WHEN t.groupID IN (419,27) OR t.typeID = 17476
THEN 0.20
WHEN t.groupID IN (26,28) OR t.typeID = 17478
THEN 0.25
WHEN t.groupID IN (25,420,513) OR t.typeID = 17480
THEN 0.30
WHEN EXISTS (SELECT * FROM eve.invMetaTypes WHERE parentTypeID = t.typeID AND metaGroupID = 2)
THEN 0.40
ELSE 0.00
end
from eve.invTypes t,eve.invBlueprintTypes b where b.producttypeid=t.typeid;

That’s everything for now. I’ll cover how I deal with price data in a later post, but it’s a little more involved, as I’m taking it all from EMDR‘s firehose, and depending on memcache.

Eve SDE SQL – Blueprint Details

5 Replies

As you may know, I’m the ‘author’ of one of the blueprint calculators that some people use in EVE Online. The one at http://www.fuzzwork.co.uk/blueprints/

If you didn’t know that, how did you get here? 😉

Anyway, it’s not uncommon to see people asking how to get the materials for building something out of the Static Data Export; and as such, I thought I’d write up a post here that I can direct people to. Oh, and if you’re looking for the code for the blueprint calculator, take a look at https://github.com/fuzzysteve/eve-blueprint-calc It’s mostly up to date, just missing a few twiddly bits.

The SQL

Now for the meat of this post. Being able to run this SQL is dependent on having a copy of the SDE up and running. If you’re using anything other than mysql, postgresql or oracle, you may have problems with the greatest function, in the first bit of sql. replace it with a case statement. You’re wanting to return 0, if the value is negative.

select typeid,name,greatest(0,sum(quantity)) quantity from (
  select invTypes.typeid typeid,invTypes.typeName name,quantity
  from invTypes,invTypeMaterials
  where invTypeMaterials.materialTypeID=invTypes.typeID
   and invTypeMaterials.TypeID=?
  union
  select invTypes.typeid typeid,invTypes.typeName name,
         invTypeMaterials.quantity*r.quantity*-1 quantity
  from invTypes,invTypeMaterials,ramTypeRequirements r,invBlueprintTypes bt
  where invTypeMaterials.materialTypeID=invTypes.typeID
   and invTypeMaterials.TypeID =r.requiredTypeID
   and r.typeID = bt.blueprintTypeID
   and r.activityID = 1 and bt.productTypeID=? and r.recycle=1
) t group by typeid,name
SELECT t.typeName tn, r.quantity qn, r.damagePerJob dmg,t.typeID typeid
FROM ramTypeRequirements r,invTypes t,invBlueprintTypes bt,invGroups g
where r.requiredTypeID = t.typeID and r.typeID = bt.blueprintTypeID
and r.activityID = 1 and bt.productTypeID=? and g.categoryID != 16
and t.groupID = g.groupID

Notes

There are three bind variables in total, two in the first, one in the second. These are all the same value, the typeid of the item being manufactured. You could just replace them, but bind variables are your friend. Learn how to use them, and love them. People that do string replacement are asking for trouble. PHP has PDO, Perl has DBI, Python has DB-API, Java has JDBC. They all support bind variables.

The reason for the two queries is that CCP have split the materials into those affected by ME, which are also recovered when you recycle it (the base materials), and those which aren’t affected (the extra materials). The base materials query here doesn’t return the materials recovered, but that’s due to the removal of materials which are marked as recyclable in the extra materials.

Extra Credit

The queries above only give you the materials for a perfect blueprint. It’s rare that you actually have one, so you’ll need to take ME and Production Efficiency into account as well. The waste factor referred to in the formulas is that one that’s defined in invBlueprintTypes. Perfect is the number of the materials defined by the above queries.

If your ME is 0 or greater, then for each material in your base materials: round(Perfect + (Perfect * \frac{(\frac{Waste Factor}{ME of Blueprint+1})}{100}))

If your ME is less than 0, then for each material in your base materials: round(Perfect + (Perfect * (\frac{Waste Factor}{100}*(1-ME of Blueprint))))

If your production efficiency is less than 5, then you also need to add (Perfect*(0.25-(0.05*Production Efficiency))) to your materials. You only do this for:

  • Base materials
  • Extra materials that also appear in the base materials

If it’s 5, you’re adding a number that’s been multiplied by 0. You may wish to do this for completeness, or so that your code works for more than just your main producer.

That’s it for now. If you have any questions, feel free to ask here, or in Eve.