Tag Archives: sde

Understanding the Eve Online SDE – Part 1

How many parts is something I can’t yet answer. As many as it takes.

The SDE is a Static Data Extract from Eve Online. It contains information like where stars and planets are, how much Tritanium goes into a Typhoon, what skills are needed for which modules and so on. It’s a useful bag of information if you’re going to be doing any development of third party tools for the game. Much (but not all) of the information is also available directly from ESI, where it’s potentially more up to data. However, it can be a lot more time consuming to process there, and some is just not available. Like what blueprints need.

The official SDE direct from CCP

I’m not going to go into a great deal of depth on this. It’s not what most people use.

From https://developers.eveonline.com/resource/resources you can download the file sde-TRANQUILITY.zip

This is the main source for the SDE, and what I use to create my own versions. The main reason most people don’t use is, is that it’s a fuckton (metric, not imperial) of yaml files. Great for passing data around, not so good for looking up something.

CCP tend to update this a wee while after a release happens. Sometimes they need poked to do it. If you’re going to work directly from it, be aware that not all yaml parsers are created equal and can handle the spec a little differently. pyYAML is probably the one to use if you can, with libyaml to stop it taking forever.

The SDE release from yours truly

For a little history, CCP used to release the SDE as a MS SQL server dump. When I got started, someone else did the conversions to mysql which I used. Then they got slower and slower, and I got impatient, finally working out how to do it myself. A few years later, CCP started converting how they did their data versioning internally, moving from what they called BSD to FSD. Which meant they moved to yaml files for much of it. At this point, I totally rewrote the converter process I was using. And because I could, I set it to target a number of different database. If I get hit by a bus, you can find it at https://github.com/fuzzysteve/yamlloader It’s python 2.7, but should be simple enough to convert to 3 if you want to.

https://www.fuzzwork.co.uk/dump/ is where I keep the conversions. I don’t promise to keep them for all time, but older versions are available if you want them. the latest version is always in the latest folder or linked to with the latest links in the root.

If you’re wondering about the structure and names, most of that is a hold over from the original SDE, so that applications needed minimal changes.

Available are:

  • MySQL, both a complete dump, and individual tables
  • SQLite, a complete database file.
  • Postgresql, pg_dump file for use with pg_restore
    • the normal one, which puts all the tables into the public schema
    • the ‘schema’ one, which puts all the tables into the evesde schema. I recommend this one for people starting out with postgres. Easier to update.
  • MS SQL server, a Data Tier Application export.
  • CSV files, now in gzip, bzip2, and plain format.

The Tables (In my conversion)

There are many tables (or CSV files) in the Eve SDE. However, these tend to fall into a small number of groups only a few of which are wanted.

The ‘Stuff’ tables

invTypes

This is the core table for many applications. If you have something which isn’t a unique item, then this is probably where you’ll want to look it up. Have a typeID from ESI which you want to look up? This is that table. Skills, ships, trade goods, generic planet types, all that good stuff. There’s a published column here which basically says whether it’s actively in the game or not.

invGroups and invCategories

invTypes has a groupID column. That links to invGroups, which has a categoryID which links to invCategories. invCategories contains a list of the types of stuff which exist. Like Ships, Skills, Planets, Modules, and so on. invGroups breaks that down further to things like Assault Frigates, Battleships, Science Skills, Asteroids and so on.

Example: you want all the ships in the game. you look up invCategories and find that ships are categoryID 6 (rather than always joining in the invCategories table.) 

select typeName from invTypes join invGroups on invTypes.groupID=invGroups.groupID and invGroups.categoryID=6;

invMarketGroups

Much like invGroups, invMarketGroups is referred to by marketGroupID in invTypes. a little complexity is added, because each market group can have a parentMarketGroupID. This is so you can have each assault frigate in a market group for their race, that market group in the assault frigates group, that market group in advanced frigates, that in Frigates, and that in Ships. (ships doesn’t have a parent). Getting the full tree in pure SQL is a complete pain in the ass.

invMetaGroups and invMetaTypes

invMetaGroups defines the various meta level groups. Like tech 1, tech 2, faction and so on. invMetaTypes links that back to invTypes so you can list all the T2 ships by joining it in. It also has the parentTypeID in case you want to see what the T1 version is.

select typeName from invTypes join invGroups on invTypes.groupID=invGroups.groupID and invGroups.categoryID=6 join invMetaTypes on invTypes.typeID=invMetaTypes.typeID and invMetaTypes.metaGroupID=2

invTraits

All those handy ship bonuses have to live somewhere. Links to invTypes for the ship type (typeID) and to invTypes for the skill (skillID). -1 is a bonus which isn’t related to a skill. unitID links to eveUnits so you know if it’s a percentage or what.

invFlags

This is related to the asset table in Eve. If you get back a list of your assets, they’ll have flags on them. You won’t see hangars and so on. The Flags determine where they are. Something with flag 11 is in the first lowslot on a ship, for example. This is why shared hangars are painful to implement.

invTypeMaterials

This _used_ to be used for making things. It’s not any more. it’s just what you get out of something when you recycle it. typeID links to invTypes for what you’re recycling, materialTypeID links to invTypes and is what you get out. quantity is what you’d get on a perfect (100%) refine.

invVolumes

When things change size as they’re packaged, the volume in invTypes becomes less useful. This is the volume of things which are packaged (when it’s different)

invItems

A table of unique items, where they are, who owns them and what they are. I don’t think I’ve ever seen anyone but CCP use this.

invNames and invUniqueNames

the names for the things in invItems. Don’t know what kind of something a thing is, but have an itemID? It might be in there. There’s almost always a better table to look at to get this information, because you normally know.

invPositions

Where the things in invItems are in space. like with invNames, it’s unlikely you’ll want to use this.

Industry Tables

When Crius came out, CCP changed up everything. And had a new yaml data file just for making shit. So I created a number of totally new tables from it. If you don’t like the format, it’s all my fault. Try and do better. I dare you.

A common column in these tables is the activityID column. This is to split them up into the various things you can do with a blueprint. Like manufacturing, researching TE, copying, reacting, inventing and so on. Yes, reactions are just blueprints like anything else now.

1Manufacturing
3Researching Time Efficiency
4Researching Material Efficiency
5Copying
8Invention
11Reactions
The Activity IDs

industryActivity

typeID is the blueprint typeid in invTypes. time is how long it takes to do that activity before modifiers.

industryActivityMaterials

Which materialTypeIDs from invTypes are used for an activityID when using a typeID (blueprint) from invTypes and the quantity before modifiers

industryActivityProducts

What quantity of productTypeIDs from invTypes are produced per run of a typeID (blueprint) from invTypes for a specific activityID

industryActivitySkills

What skillID from invTypes is required to do activityID on a typeID (blueprint) from invTypes and what level it needs to be.

industryActivityProbabilities

Invention has different probability depending on the typeID of the blueprint (invTypes) with a handy productTypeID to work out what it makes.

industryBlueprints

maxProductionLimit is how many blueprint copies can be on a BPC.

Planetary Interaction.

planetSchematics

The various things you can make on a planet and how long they take.

planetSchematicsPinMap

Just links the schematics to the kind of structure on planet which makes them. Like basic industry. though it is by planet type as well.

planetSchematicsTypeMap

Takes the schematicID from planetSchematics, and tells you the quantity of what typeIDs you need to put in (isInput=1) to get typeID out (isInput=0)

The Dogma Tables

Dogma is Eve’s engine for knowing stuff about things. What slot does a module go into? Dogma. What skills are required for something? Dogma. How much damage does a gun do? Dogma (though it’s hard to work out. yay)

dgmAttributeTypes

The table with the description of the various dogma Attributes. by attributeID

dgmTypeAttributes

The table which links dogma attributes attributeID to item types invTypes typeid. These days, just use valueFloat for the value. everything is condensed into that now. Handy for things like how many low slots something has. Or which skills are required. (with a separate attribute for which level it’s needed at. yay)

To see all the attributes and what they do, for a Rifter (typeid 587) try:

select * from dgmTypeAttributes join dgmAttributeTypes on dgmTypeAttributes.attributeID=dgmAttributeTypes.attributeID where typeID=587;

dgmEffects

The various bits of dogma which are just flags on items, rather than things with values. like if it’s a low slot module or not. This describes those flags.

dgmTypeEffects

Linking typeID from invTypes to effectID in dgmEffects

dgmAttributeCategories

Just splits the various attributes up into categories. Like if it’s a fitting attribute, shield, or so on. categoryID in dgmAttributeTypes

dgmExpressions

This is stuff which relates to how things actually work in eve. the code behind it. I don’t understand it well enough to explain it.

Map Tables

mapDenormalize

This has pretty much every celestial in it, and where it is, in meters, in its star system, with the star at 0,0,0

Links to invTypes so you know what some is (planet etc) by typeID and invGroups by groupID. links to mapSolarSystems with solarSystemID, mapConstellations with constellationID mapRegions with regionID.

What might not be obvious is wormhole effects are here, with a groupid of secondary sun.

mapJumps

Less useful than it might sound, it tells you which pairs of stargates link up. You probably want mapSolarSystemJumps

mapSolarSystemJumps

what solarSystemIDs have stargates to what other solarSystemIDs. also has regions and constellations

mapSolarSystems

So you can find out information about solarsystems by solarSystemID. Names, regionIDs, ConstellationIDs, where they are in the universe in meters. (yes, light year ranges stored in meters) and their security rating.

mapRegions

Simiar information, but for Regions.

mapConstellations

Take a wild guess 😉

The end, for now

There are still a bunch of tables which I’ve not talked about yet. Most are pretty self explanatory, but I’ll get to them another time. This should be enough to get you going.

Useful bits and pieces – Eve SDE

4 Replies

Once you get comfortable with the CCP way of doing things, the SDE isn’t too bad to work with. You’ll have to dig around in it to get some of the numbers you want, and sometimes it’s the opposite way round from what you’d expect, but it’s not too bad, most of the time. Here are a few more useful snippets for getting information on things.
Continue reading →

Eve SDE SQL – Blueprint Details – Part 2

3 Replies

I covered the basics of Blueprint details from the SDE back in my last post, which covers most of what people actually want to know. There are a few extra bits which the more dedicated coders will want. They’re simple enough to work out, but it’s time you might not want to have to look at. Consider this a round up post.

Perfect Blueprints

Perfecting blueprints is, in many cases, a waste of time. You want ‘good enough’, rather than perfect. It’s also unattainable for some. A Raven would take you over 4 million days to perfect. And half of that time (or so) would be to save a single Trit.

I don’t have a formula for a ‘good enough’ level. It’s very dependent on the types of materials being used. The Formula for perfect, for each material, is:

floor(Quantity*(\frac{(\frac{Waste Factor}{100})}{0.5})

If you want the perfect for an entire blueprint, pick the material with the highest quantity.

As before, the Waste Factor comes out of invBlueprintTypes.

Production Times

More details from invTypeBlueprints.

If the PE of the blueprint is 0 or above, the total time in seconds is:

productionTime*(1-(\frac{productivityModifier}{productionTime}*\frac{Blueprint PE}{1+Blueprint PE}))

If the PE of the blueprint below 0, the total time in seconds is:
productionTime*(1-(\frac{productivityModifier}{productionTime}*(Blueprint PE-1))

To then adjust it for your industry level

Time from Above * (1 - (0.04 * Industry))

So to recap, the time in the database is the maximum time it’ll take to manufacture something (assuming it’s not a negative PE), with everything else reducing the time. The changes also fall off rapidly. PE beyond 5 or so is often worthless.

Materials to invent off of something

select t.typeid,t.typeName, r.quantity
from ramTypeRequirements r,invTypes t,invBlueprintTypes bt 
where r.requiredTypeID = t.typeID 
and r.typeID = bt.blueprintTypeID
and r.activityID = 8
and bt.productTypeID=?;

The bind variable is what you’re making. You could simplify the query if you had the blueprint ID already, eliminating the join to invBlueprintTypes, going straight to ramTypeRequirements. It’s possible to pull the damage too, but the SDE currently says the Interface is used up.

If you want to eliminate the interface from the pulled materials, use this sql. It’s eliminating interface, by ignoring the group it belongs to. To pull just Interface, then turn the != into a =

select invTypes.typeid,invTypes.typename,ramTypeRequirements.quantity
from ramTypeRequirements,invBlueprintTypes,invTypes 
where producttypeid=?
and ramTypeRequirements.typeid=invBlueprintTypes.blueprintTypeID
and activityid=8
and invTypes.typeid=requiredTypeID
and groupid !=716;

Skills to manufacture something

select t.typeName Name, r.quantity Quantity
from ramTypeRequirements r,invTypes t,invBlueprintTypes bt,invGroups g
where r.requiredTypeID = t.typeID
and r.typeID = bt.blueprintTypeID
and r.activityID = 1
and bt.productTypeID=?
and g.categoryID = 16
and t.groupID = g.groupID

The bind variable is what you’re making. You could simplify the query if you had the blueprint ID already, eliminating the join to invBlueprintTypes, going straight to ramTypeRequirements.

Skills to invent something

This is a little bit of a pain. The skills are tied to the datacores and interface, rather than the job. Specifically they’re in dgmTypeAttributes, under the attributeIDs of 182 for the skill, and 277 for the level of the skill. As they’re done that way, pulling it with a single sql query is somewhat difficult, with a pivot table required (use if conditions, along with a max, then group them). When I’ve done it, I’ve just used 2 queries, and handled the pivoting in the code.

The skills

select t.typeid,t.typeName skill,r.requiredTypeID
from ramTypeRequirements r,invTypes t,invBlueprintTypes bt,
dgmTypeAttributes dta 
where coalesce(dta.valueInt,dta.valueFloat)= t.typeID
and r.typeID = bt.blueprintTypeID
and r.activityID = 8
and bt.productTypeID=?
and dta.typeid=r.requiredTypeID
and attributeID=182

The requiredTypeID refers to material you need the skill for. In this case, datacores and interfaces. Everything else is the skill.
The bind is what you’re making.

The skill levels

select r.requiredTypeID,coalesce(dta.valueInt,dta.valueFloat) level
from ramTypeRequirements r,invBlueprintTypes bt,dgmTypeAttributes dta
where r.typeID = bt.blueprintTypeID
and r.activityID = 8
and bt.productTypeID=587
and dta.typeid=r.requiredTypeID
and attributeID=277

The requiredTypeID refers to material you need the skill for. Level is the minimum level for the skill.

In the previous 2 queries you’ll have seen the use of coalesce. That’s because CCP seem to have had trouble deciding which to store the value in. It’s normally the valueInt, but not always.

Research times

researchMaterialTime and researchProductivityTime are straight out of invBlueprintTypes.

Material Efficency research

Per level:
researchMaterialTime*(1-(Your Metallurgy*0.05))

Productivity Level research

Per level:
researchProductivityTime *(1-(Your Research*0.05))

Times are all in seconds.

Getting Basic Blueprint Information

select productionTime,wasteFactor,productivityModifier,researchProductivityTime,researchMaterialTime
from invBlueprintTypes
where productTypeID=?;

This is most of the information needed in previous equations.

Is this obtainable through invention?

select parentTypeID
from invMetaTypes
where typeid=?
and metaGroupID=2

This will only return a result if you can invent the blueprint from another one. The parentTypeID is the typeid of the original type. Such as a rifter, if you’re looking at a wolf.

Chance to invent

This isn’t actually in the SDE. However, I found some handy sql to stick it into an table you can use for lookups. The type id is that of the product, rather than the blueprint.

create table inventionChance(typeid int,chance float);

insert into inventionChance (typeid,chance)
select typeid,CASE
WHEN t.groupID IN (419,27) OR t.typeID = 17476
THEN 0.20
WHEN t.groupID IN (26,28) OR t.typeID = 17478
THEN 0.25
WHEN t.groupID IN (25,420,513) OR t.typeID = 17480
THEN 0.30
WHEN EXISTS (SELECT * FROM eve.invMetaTypes WHERE parentTypeID = t.typeID AND metaGroupID = 2)
THEN 0.40
ELSE 0.00
end
from eve.invTypes t,eve.invBlueprintTypes b where b.producttypeid=t.typeid;

That’s everything for now. I’ll cover how I deal with price data in a later post, but it’s a little more involved, as I’m taking it all from EMDR‘s firehose, and depending on memcache.

Eve SDE SQL – Blueprint Details

5 Replies

As you may know, I’m the ‘author’ of one of the blueprint calculators that some people use in EVE Online. The one at http://www.fuzzwork.co.uk/blueprints/

If you didn’t know that, how did you get here? 😉

Anyway, it’s not uncommon to see people asking how to get the materials for building something out of the Static Data Export; and as such, I thought I’d write up a post here that I can direct people to. Oh, and if you’re looking for the code for the blueprint calculator, take a look at https://github.com/fuzzysteve/eve-blueprint-calc It’s mostly up to date, just missing a few twiddly bits.

The SQL

Now for the meat of this post. Being able to run this SQL is dependent on having a copy of the SDE up and running. If you’re using anything other than mysql, postgresql or oracle, you may have problems with the greatest function, in the first bit of sql. replace it with a case statement. You’re wanting to return 0, if the value is negative.

select typeid,name,greatest(0,sum(quantity)) quantity from (
  select invTypes.typeid typeid,invTypes.typeName name,quantity
  from invTypes,invTypeMaterials
  where invTypeMaterials.materialTypeID=invTypes.typeID
   and invTypeMaterials.TypeID=?
  union
  select invTypes.typeid typeid,invTypes.typeName name,
         invTypeMaterials.quantity*r.quantity*-1 quantity
  from invTypes,invTypeMaterials,ramTypeRequirements r,invBlueprintTypes bt
  where invTypeMaterials.materialTypeID=invTypes.typeID
   and invTypeMaterials.TypeID =r.requiredTypeID
   and r.typeID = bt.blueprintTypeID
   and r.activityID = 1 and bt.productTypeID=? and r.recycle=1
) t group by typeid,name
SELECT t.typeName tn, r.quantity qn, r.damagePerJob dmg,t.typeID typeid
FROM ramTypeRequirements r,invTypes t,invBlueprintTypes bt,invGroups g
where r.requiredTypeID = t.typeID and r.typeID = bt.blueprintTypeID
and r.activityID = 1 and bt.productTypeID=? and g.categoryID != 16
and t.groupID = g.groupID

Notes

There are three bind variables in total, two in the first, one in the second. These are all the same value, the typeid of the item being manufactured. You could just replace them, but bind variables are your friend. Learn how to use them, and love them. People that do string replacement are asking for trouble. PHP has PDO, Perl has DBI, Python has DB-API, Java has JDBC. They all support bind variables.

The reason for the two queries is that CCP have split the materials into those affected by ME, which are also recovered when you recycle it (the base materials), and those which aren’t affected (the extra materials). The base materials query here doesn’t return the materials recovered, but that’s due to the removal of materials which are marked as recyclable in the extra materials.

Extra Credit

The queries above only give you the materials for a perfect blueprint. It’s rare that you actually have one, so you’ll need to take ME and Production Efficiency into account as well. The waste factor referred to in the formulas is that one that’s defined in invBlueprintTypes. Perfect is the number of the materials defined by the above queries.

If your ME is 0 or greater, then for each material in your base materials: round(Perfect + (Perfect * \frac{(\frac{Waste Factor}{ME of Blueprint+1})}{100}))

If your ME is less than 0, then for each material in your base materials: round(Perfect + (Perfect * (\frac{Waste Factor}{100}*(1-ME of Blueprint))))

If your production efficiency is less than 5, then you also need to add (Perfect*(0.25-(0.05*Production Efficiency))) to your materials. You only do this for:

  • Base materials
  • Extra materials that also appear in the base materials

If it’s 5, you’re adding a number that’s been multiplied by 0. You may wish to do this for completeness, or so that your code works for more than just your main producer.

That’s it for now. If you have any questions, feel free to ask here, or in Eve.