Over the past week I’ve been setting up a new server, tweaking the settings and getting the site to run as fast and efficient as it can do in it’s current codebase. Tonight I started migrating the data off the old server at 21:00 UTC and blake changed the DNS to point to the new server. It took about an hour for the new domain IP address to populate but I can happily say that the site is now currently operating fully and back open for business at the same address (http://industry.darkshadowindustries.com).
Of course if there’s any problems that I haven’t foreseen effecting you then please let me know asap so I can look into it.
Current development of this tool has been slow recently due to the rebuild of staticmapper but also because work keeps throwing things at me to do. But one thing I can do is try to alleviate some of these slowdowns that keep happening.
To get you all into the picture, the site is currently hosted on a shared host vps. This appears to be a single server running multiple vps. The tracker is very resource intensive and I’ve done the best I can to try and minimize it’s foot print, but due to the nature of the EMDR consumer and all the api requests it has to deal with ontop of day to day traffic, the vps is finding it’s self fighting for Input/Output resources (that’s reading and writing to the hard disc and memory for any none techies) with other vps services on there and when things get too bad they bounce (restart) our vps. Generally this means me getting up in the morning to lots of automated emails from the site telling me mysql tables have crashed, mongo caches need rebuilding and other general mayhem that make my day oh such a pleasure sometimes.
The only way to fix this is to move the site to another solution. I will be moving the site next weekend to a Digital Ocean droplet. Because DO are cloud based we (hopefully) won’t suffer from these I/O related blackouts and other hiccups that happen from time to time. I am currently testing a slightly different configuration of server for the move which will make the redevelopment easier. Currently the tracker runs on a ubuntu 12 LAMP stack. A fairly basic default web server. The new server I’m testing with is running Centos 6 and instead of Apache, I’ll be moving forward with nginx. Lots of reasons for this which I may blog about later, but bench testing the difference between my local vagrant ubuntu apache and the centos/nginx setup shows that the nginx box is marginally faster and better with it’s resources than the apache one.
The move will be happening at around 23:00 UTC on the 24th Jan. That’s next friday and will be out for approximately 3-4 hours. This should give me enough time to migrate the database and run some last minutes tests to make sure that all is still well and I haven’t missed anything. Re-pointing the DNS of course may take longer, though these days it tends to happen in minutes.
So put that date down in your diaries
The biggest feedback request from the latest bunch of EMDR related market updates was graphs on the market feed. So heeding the call, I’m now collecting a 24 hour snapshot of prices as they evolve on a second by second basis. Basically every time a legitimate EMDR update is detected and displayed on the market feed, the bid and ask prices are also captured, saved and kept for 24 hours to give you a running indication of how the prices have risen and fallen over the previous day. This is another utilization of nosql caching in mongodb so data retrieval is lightening fast.
To view the charts, click on the expander arrows next to each mineral icon and the charts will be inserted in a row beneath. As new prices are fed into the market feed, the charts will also update dynamically in real time. I’ve opted to use highcharts so you can zoom into the displayed data, or click on one of the 3 preset options available.
Because the price information is displayed in real time it won’t always display information for all 4 hubs on the tooltip. The graph is configured to show the actual data points when prices are collected so you can also see when are the most active times and markets.
The question has been raised as why I picked Hek and not Rens. At the time I figured that Hek was the biggest trade hub but on actually interrogating the data I found that Rens has almost twice the market order as Hek so I’ve change the Minmitar station.
Following on from the work I did last week with the websockets and front page dashboard items, I’ve taken it the next step and produced a mineral index page showing all 4 main racial trade hubs and current prices in said stations.
At a glance you can see the price difference in trit between Jita and Amarr and also keep an eye on what the market is currently doing. Prices are updated in real time and if you mouse over each cell you can get further information about the order such as when it was issued, it’s duration, when it was collected and sent by the client, when it arrived at the consumer and the latency (delay) it took for the information to arrive.
Prices will change from green (price going up), red (price going down) and blue (no change to the price since last update). Changes in price will flash the back ground yellow letting you know when changes in the market are going on. Please note though that the updates won’t work on the IGB at present. This is because the in game browser doesn’t support web sockets at present, so page updates will need to be manual for now. I could add an ajax type update (like that currently still on the dashboard home page for IGB and non websocket browsers) if there is demand for it.
The “live” index can be viewed here and of course the obligatory screeny
I will be expanding on this feature over the next week or so. The next addition will be a 24 hour track of the prices with nice line graphs showing price trends. Probably a scatter graph as well to show how many updates are processed during peak and slow hours. I’ll also be plugging in the rest of the site mineral price lookups into the live mongo cache which will speed things up a bit more.
Past couple of weeks have seen me getting friendly with HTML5 web-sockets and a rewrite of some core EMDR market based features. The result is that I’ve completely rewritten the method that mineral index price charts on the home page get updated. Previously, the delay between price data being accepted by the site’s consumer, getting bulk inserted into the database and displayed into the front page could take as much as 4 or 5 minutes. Ok that’s not a massive amount of time but I find that nothing is as good as hot off the press live as you can get it market data. The consumer has been tweaked to broadcast mineral prices at the 4 main racial trade stations to any connected client sockets the moment they arrive. The previous ajax pod update method is still available as a fall back for non web-socket browsers such as the IGB and some tablet browsers.
This is not by any means the end of it. I’m currently developing my own customized web-socket framework that will be incorporated into other areas of the site. I can see for instance a massive advantage in the project section where an enormous amount of information and calculations need to be carried out before the final data set is presented to the client as a JSON object. A webs-socket will enable smaller chunks of data to be passed on an event driven model as and when you request. With the correct authorization security between socket and session, any API updates could be presented the moment it’s completed and this is just the start.
I’m also planning a cache model to store some price information using mongo. There is a lot of price lookups on the site so to be able to cache the most frequently requested items (such as minerals) will save further work on the mysql database resulting in a faster responding site.
If you want to see the feed in action simply open the home page and wait for the magic to happen. It won’t take long. Registered users can add the other 3 main trade hubs to your homepage dashboard as a configurable pod so you can watch the whole market. This weekend I’ll be working on a market information page to properly display the current market, updated in real time with graphs, useful info etc.
As many of you know, the API has it’s moments and this month has certainly seen a lot of instability. This generally makes for a lot of eve mails notifying me that the API is no longer updating unfortunately there’s not a lot that I can do when it’s CCPs side that’s giving the errors.
So to help people see what’s going on behind the scenes, I’ve added an API statistics monitor. there are 3 charts, current day, 30 day and whole year. If you see a spike of errors and the API isn’t updating then you’ll now know why. The current day is a live chart showing current information. It will update every 15 seconds so you can see in realtime any errors that we’re experiencing.
A couple of people have mentioned the possibility of me adding a force update page for corporate APIs like you have for personal APIs in your character admin area. Due to the sheer volume of corporation requests the site is currently making, I do not think this is a good idea. The API works by specifying a cache time when the next update can be carried out. The site uses a series of job daemons to asynchronously start corporate updates within seconds of the timer expiring automatically. If your API is not updating it will be because of errors on CCPs side so a manual update will only exacerbate the situation. I have mechanics in place to work in accordance with the results of those errors and adding a force update page would open up the possibility of swamping the API server with needless requests. Too many unwarranted requests and CCP place the site with an IP ban.
A feature I’m planning right now are notification icons to appear in the footer if an error is thrown by the API. A tooltip will display any information regarding the error along with (depending on the type of error thrown) links to various parts of the site or CCPs site where the error could be remedied.
When I originally “fixed” this, I’d made sure that it was working, well working for me at any rate. I do all my industry through my alt corp with corporation projects and hadn’t actually checked that I’d made the same changes to the personal side of things. So due my failing QA standards I pushed the half completed code thinking all was good and done. Unfortunately this left some of you out of the loop and for that I do appologise. Some important RL work slowed down this bug fix so it’s been sitting there still broken for the past couple of weeks 😦
Good news though is this is now fixed and verified so you can now add personal assets to personal projects.
As the title says, adding assets to projects was something I broke on the revamp of the projects section but never got round to fixing. There was a reason for this, namely it was using parts of the asset section which I found were a little bit incompatible with how I’d reimplimented the projects, but this week I’ve rewritten those parts and added this important feature back into the projects view.
You can access it on the Assigned Resources tab and selecting Assets in the Add Resource section. After selecting the station/location, you can check the tick box which opens up a dialog window allowing you to set how much (up the maximum actually needed resource) you want to assign to the project.
One of the better recieved features recently was the addition of the time line to projects and current ship manufacture indicator on the industry jobs page. They’re good indicators of what’s currently going on and how you’re performing on a specific project.
Recently I found myself wondering on how efficient I was being with some of my BPOs. Any of you following Blake’s blog will know that we’re heavily into carrier industry and I try to keep my BPO sets in operation at all times but there are patches, either due to logistics, material shortages or just time when they’re not and I wanted to see if this could be displayed in a gantt style chart. So I set to tweaking the time line to this. I am very happy with the result.
The filters allow you to set a time frame, activity type, categories, groups and a type name. Even some various logic in there in case you want to search for items containing {your_search_string} in it’s name.
Be cautious though if running a query that could potentially bring back a lot of results. The initial search is fast but it can take a while to render the chart. All the processing is client side so how long it takes depends on your computer power but even my gaming rig with an overclocked i5 took about 2 minutes to render my full personal industry with something close to 20k individual jobs accross my alts.
You can access the filters from the industry jobs section at the top of the page.
This update will effect mainly those of us who operate in null or lowsec and move large quantities of minerals in one go. The best method of doing this is with compression. Only problem with compression is that the levels of materials you need to compress all the trit and pyerite rarely line up with your project need so you always have to end up buying more of things like isogen and pye if you’re using 425mm railguns like most people do. I usually keep forgetting how much extra isogen I’ve used so currently I have an excess of isogen that means I probably don’t need to buy any for a couple of months now. Unfortunately it’s all up in our production system. damn. This widget will take the pain out of those calcs and help you decide what needs to come back from production to go into the next round of compressions. Eventually you should have the same number of isogen and pye going round and round the system meaning you only ever have to buy the project requirement.
A new tab on a project’s view lets you specify a blueprint to work from, the ME, PE for time and with a pivot mechanic lets you pick which mineral to base the compression on. You can also decide to go with a full compression need or just for the unassigned minerals. The result will tell you how how many units of compressed item you can make, what you will have left over or how much extra you need to buy in order to make it work, how much will be lost on reprocessing (not including NPC/sov holder tax) and most imporantly, the M3 volume you will end up hauling with your loose ends and compressed items. Extra materials show up as 0M3 as those minerals will in fully used up in the compression.
* Edit *
I’ve added a mechanic to allow you to enter in multiple blurprints at a go and select between those as well to find the best compression ratio.
