No one enjoys database backups. They usually involve a load spike and a lot of table locking (even in best-case scenarios) which can be felt on live servers. Some sites bring their services down to perform backups, others slow to a halt. It is a very important aspect of running an online service, and finding an optimal and elegant solution is usually very specific to the infrastructure and nature of services being offered.
I am very serious about keeping live backups. The osu! database is replicated to a slave server, providing a real-time fallback in case the main server happened to fail. This is already ample to handle any server software/hardware issues – for instance if a drive was to fail. This alone does unfortunately leave the possibility of human mistake – where the database is damaged internally – which while I’d love to say doesn’t happen, is generally unavoidable (especially when expanding the size of the team which is working with database access). In the case of a human mistake, both the master and slave server’s data is in a bad state, making slaves of this nature useless.
In the case of human error, my current backup solution was to take database snapshots from the slave server. This results in very minimal effect on the front-facing service, as user based actions rarely require a query to the slave database, but the storage requirements, the IO requirements and the general clunkiness of snapshots has always bugged me. It also means that as backups were only made once a week, in the case we needed to recover data it may be up to seven days old, which is not acceptable.
Incremental snapshots is one way to avoid this pitfall, but does require all database tables to be of InnoDB engine. I regularly test InnoDB (or in this day and age, XtraDB) but am still getting better overall performance with the arguably less reliable MyISAM, so this is not an option.
Introduce a delayed slave to the equation. This is a separate server which is initialised as a slave to the master database, but maintains a time distance from the live data. This is easily done using the Percona Toolkit’s pt-slave-delay, which runs as a daemon and allows specifying a period for which it should delay sql operations by.
There are a few amazing advantages here:
To initialise the pt-slave-delay command, it’s as simple as ensuring replication is started, then specifying the delay and check interval. I am currently using the following, which should be run at system startup if you want it to persist:
#!/bin/sh
mysql -e 'start slave;'
pt-slave-delay --delay 24h --interval 5s --no-continue localhost
Take note that while replication is stopped, you will now be able to see how many seconds behind the server is using the SHOW SLAVE STATUS command. As I regularly use this for monitoring the slave delay, I had to use an alternative method to find the delay. For me, the easiest way was to select the MAX(timestamp) from a table with high activity and compare this to CURRENT_TIMESTAMP as follows:
SELECT UNIX_TIMESTAMP(CURRENT_TIMESTAMP) - UNIX_TIMESTAMP(max(`date`)) AS seconds FROM `osu`.`osu_scores`;
I really enjoy databases and the optimisation of them at low and high levels. osu! is still relatively simple when it comes to database infrastructure but it is rapidly expanding. Keeping up with the increasing load is an interesting and very fun process. I hope to post more articles like this delving into the slightly more technical side of things going forward.
Update:
I just found out that as of MySQL5.6 (which I am actually running, so have switched to this method) you no longer need the pt-slave-delay script as this is built-in functionality. You can add a delay with one simple command (make sure to STOP SLAVE; first):
CHANGE MASTER TO MASTER_DELAY = 14400; --delays 4 hours
p.s. I haven’t forgotten about the “this week in osu!” series, but some of the things i planned on writing about have been lost in my forgetful mind. I’ll try and knock one out along with the next public release, which I am hard at work on getting finalised. I am trying to livestream as much as I can, so if you are interested in the development of osu!, make sure to tag along and say hi in chat :).
I recently tweeted about a life-hack article titled “The distraction-free iPhone (or ‘Why I’m happier since I disabled Safari’)”. The general gist is that our phones are taking over our lives, and acting as not only a procrastination device, but a distraction from what is going on around us.
“Checking email, checking Twitter, checking news. Wondering if something interesting was happening anywhere in the world. Wondering if anybody was thinking about me.”
Although I love how powerful our phones have become, this is definitely something I have started to feel over the last couple of years. It is especially disturbing when going out to lunch with fellow software engineers and only making conversation when someone comes across an interesting tweet, or an amusing image on instagram. This consumption of information is somewhat of an addiction, and while I tend to focus on consuming only what is relevant from the business side of things – I avoid the social aspects of twitter and similar services – am still guilty of indulging in it.
On reading this article I decided it was definitely a time for a change. Due to my commitments to osu!, osu!stream and puush, I knew that I could not completely lock myself out of services which brought incoming real-time communications which could require immediate response, so after some very deep consideration I began to form an action plan. I felt it was worth writing up exactly what I have decided on, and my reasoning behind each decision.
So, this is my iPhone’s only home screen:
Using Springtomize (a jailbreak tweak) I am able to hide not only icon labels, but any apps which I don’t want appearing on my home screen. I am also able to disable spotlight and pagination completely. Using this technique, I am able to hide Tweetbot and Facebook while still receiving incoming notifications via notification centre. There is no way to access these application (which would be the main sources of information) as long as I do not receive incoming communications.
Let’s run through each of the apps I have left visible and installed, from top-left to bottom-right:
You may be wondering where system setting are. I actually have a few more apps hidden in an invisible folder, which are utility-class apps that are necessary for maintenance of the phone itself (Cydia / App Store / Settings) or useful in emergency situations (iTap Remote Desktop / Prompt SSH / Dropbox). Out of sight, out of mind, until the day they are necessary.
So far I have not looked back. It sure does limit what I can do with my phone, but I don’t see this being an issue. I have already noticed a reduction in the number of times I mindlessly reach for my phone when I don’t actually have a particular task I need to perform.
I urge others to give this a try. You don’t need to go the full way (like Jake in the original article) but why not try cutting back what you use your phone for, and spending more time paying attention to what is going on around you; interacting with those that are close to you; taking in more of life. We already spend so many hours in front of a PC screen, after all.
Been working so hard on new stuff that I’ve basically forgotten all the things I have done and need to post about. Let’s get started with the big one: the osu! UI and default skin graphics refresh!
Over the course of two long weeks, I spent basically every minute awake in photoshop going through all 421 osu! sprites and doubling the dimensions of each and allowing pixel perfect display at 2048x resolutions. Yep, osu! is now “HD”, if you will! Where I was no longer happy with the old design, or where vector data was not available, they were re-created from scratch. This ended up being over half of the resources.
Let me start by saying that I couldn’t have done this alone. A huge thanks to RBRat3 for helping out with a lot of the icons which did not have vector data, including (but not limited to) the main menu icons, the editor control icons and the mod icons.
I also took this opportunity to not only split the resources into two separate resource collections (UI and gameplay), but also decided to distribute in the compressed PNG format rather than raw bitmaps. This is a trade-off allowing for faster patching and less disk usage while adding a load-time performance overhead. This overhead worked out to around 5% and does not affect runtime performance – plus it is only relevant when using the default skin, so all the people using custom skins won’t even notice a difference – so I figured it was beneficial overall.
I will be distributing the elements of this skin for use as templates and/or modification in the near future. The only reason I haven’t yet is because there are a few parts left which are not finished, including Taiko / CtB / osu!mania modes. Keep a watch on the skinning forum for the template release in the near future.
BUT WAIT, this update didn’t just stop at sprite changes! While changing all these sprites, I found so many other things along the way that I was no longer happy with, and took the liberty of changing them too. Let’s try and make a list!
The follow points (those dots which display between hitobjects) are now directional. In the new default skin, they become arrows which – in my testing – really help with map readability.
Hit explosions are visually much smaller, allowing increased readability and reducing the cases they overlap future hitobjects. They also have a really nice glowy effect and a true particle system!
You might also notice that they will briefly appear top-most, but quickly fade to a new position underneath hitobjects, helping with readability on streams.
The ranking popup dialog which used to show your online ranking has been moved so it is no longer a popup. It is now accessible by scrolling downwards, or clicking the handy button at the bottom of the screen.
Special mode selection screen is gone. You can now change modes from song select using the new “Mode” menu! This makes a lot more sense to me these days, where the special modes are basically treated the same as osu! mode.
Scrollable views can now be dragged. Also scrolling is a lot smoother, with improved acceleration/inertia handling (please don’t sue me for patent infringement apple <3).
The back button has been unified across the game (it used to behave slightly differently in some cases).
The background has been removed from osu!direct. It now uses the default menu-background or a user replacement if one exists.
I’ve probably forgotten some other stuff, but I have a few more posts up my sleeve, so will tag it on the end of one of those if I do happen to recall it. I’ve already received quite a lot of feedback on the new design, but feel free to leave anything you have to say in the comments here as well :).
just as an interim update, i have been super-busy making awsome and decided to continue making awesome rather than post about it. once things are done i will make a couple of posts to make up for it.
Catching up on bug-fixes, I have finally been able to work on a few new features which are visible to the end user. Let’s get right into it!
The user panels display now adjusts to display four columns on all widescreen modes. This solves the long time problem of having a gap to the right on some widescreen aspect ratios.
It also allows for over four columns on super-widescreen resolutions, should you be running across two monitors using something like eyefinity.
osu!mania now support vertical flip, making gameplay similar to that of DDR/o2jam. This should help people coming to osu! from said game systems adapt quite easily, instead of resorting to flipping their monitors upside-down. I was planning on adding this one, but woc beat me to it ;).
The skin you see above is available here.
Skinning now supports double-resolution sprites, allowing for very high resolution gameplay. osu! now uses these sprites when the window width is greater than 1600 pixels.
To add support to a skin, add images with double the width/height containing the “@2x” suffix before the file extension.
The editor now supports widescreen resolutions more elegantly. The biggest change is allowing selections outside the play-field grid. This works even when the current placement mode is not select, meaning you can start a selection while in circle or slider placement mode as long as you begin the selection outside the grid area.
I completely restructure Bancho to allow for usergroup authentication on a per channel basis. In the process, I was able to combine the osu!stream and developer channels into the main bancho server, reducing the overhead to get people into the chat rooms. In the future, this also means we can have chat rooms pop up for teams in the case of tourneys or similar. Much more flexibility :).
Got more awesomeness already in the works. Stay tuned!
p.s. Sorry for the slider issues in the editor. Hopefully all the bug fixes made up for it.
Yet another busy week (and a bit) still catching up on support queries, combatting increasing amounts of cheating and putting some extra time into osu!stream. Making good forward progress, though!
The in-game chat FAQ system has now been moved to the more manageable google docs format. This means that anyone can contribute both FAQ entries and localisations to other languages. Currently you can query specific languages by adding the language code before your !faq command.
!faq ask will return the first language found (defaulting to english)
!faq it:ask will return the same faq entry in italian, but fall-back to english if not found.
You can find more details about contributing and localising in the osu! localisation project thread.
BanchoBot now has a profile! Keep a watch as we build this up into something which will hopefully help people harness his power.
I thought I had more to say, but it seems these are the only recent changes that have been made which are of a nature I can make public. I can say that I am now up-to-date to the point I can hopefully clear out the open bug tickets and push some new features I’ve been working on out into the public :). Stay tuned.
I’ll make no excuses for my lack of updates. Let’s move straight on to what has changed over the last couple of weeks!
The osu! website now makes use of SSL for all communications after logging in. This happens on the next login attempt, so if you are using a stored cookie, you may need to re-login once to get switched over.
I am providing ssl support via a cloudflare issued certificate on a trial basis. My reasoning for doing this is that they (via globalsign) host the fastest SSL resolution available, beating the competition by ~50-80% depending on location, and allowing for almost no overhead in SSL negotiation time.
As a result, the whole site is also being served via cloudflare’s piping service. Looking at analytics since enabling this, there has been negligible measurable change to performance (client-side load time), so it looks to be a keeper. It also allows me to throw up a custom error page should there be connectivity issues with data-centres or servers, which came in quite handy with recent data-centre troubles.
Speaking of data-centre troubles, I am currently monitoring the situation after the downtime experienced over the last month. The issue was quite a serious one, and while I don’t believe Softlayer’s resolution was completely correct, they were trying hard to get things back online.
The one area I am not happy with is the delay in communication from their end, and the lack of public acknowledgement (all ticket details are available only to clients via private support tickets). I will be considering my options, but likely won’t move from Softlayer this time around. They have a very stable network usually, and this may likely be the first major disaster at their end, so I am willing to give them a second chance.
woc2006 has been working hard on a new version of the osu!mania editor (let’s pretend the last one didn’t exist). It has a proper osu!mania layout and controls which allow for both real-time and manual mapping. Currently available on the test build for those who wish to check it out!
I pushed fixes to taiko backgrounds, by giving taiko mode a “metadata” bar below the playfield. This displays common information like the song artist/title, and replaces custom-baked backgrounds which were becoming quite prominent in taiko maps.
In order to make this work correctly, I shifted all taiko backgrounds based on the standard “taiko map template”, which not only provides a better default alignment, but makes sure the metadata in the background is not visible. The taiko playfield is no longer transparent, so any custom skins should be updated to take this into account and avoid artifacting.
The font used on the “metadata” bar defaults to a system font, but can be overridden on a per-skin basis by adding a taiko.ttf file to your skin. The taiko skin has been updated to reflect this (and should be automatically downloaded by the updater if you previous grabbed it via this method).
Oh, and the taiko playfield bar now fades in and out of kiai time :).
The osu! wiki has now been integrated with the main site design, and the old FAQ decommissioned as a result. The transition isn’t complete yet, with some styling looking a bit off and some basic javascript not working, but it is much better than what we had before. I hope this will
Stay with me here. I’m trying my best given circumstances.
The observant will notice that I have been a bit less active over the last couple of weeks. This is partly due to the fact that I have been moving/settling in to a new living and office environment, and catching up with a lot of people I haven’t seen in a while. Things are getting back on track though, and I have been busy not only working on my own new features/fixes, but also merging the contributions from Ephemeral and woc2006.
Added the ability to reset key bindings to defaults. This is one of those feature requests I see and instantly wonder why it isn’t already implemented. I then proceed to drop everything and implement it immediately.
Fixed remaining issues with “Loading…” players in multiplayer games. This was being caused due to a constant not being the same on server and client, meaning if a certain number of presence requests were made at once, they would be silently ignored by bancho (causing those players to remain in a loading state forever).
Widescreen support in Match Setup thanks to Ephemeral. This means that as far as I can tell, every game mode/screen in osu! is now widescreen compatible. Quite a good achievement!
A few futher tweaks were required to hyperdash after getting feedback from pro players. This was an interesting one to follow up on, because while a lot of players voiced how much they disliked the new changes, they could not say why. Luckily I did find someone who could pinpoint the issue at hand – specifically the behaviour of the “dash” button during hyperdash – and it will be pushed out as a fix in the next release.
I did a lot of internal clean-up on error handling scenarios, to make processing error reports a lot less painful. Also fixed a few scenarios where osu! wouldn’t automatically restart after it says it will.
“Finally” updated the main menu copyright graphic, which is no longer really that. I removed the (in my opinion) unnecessary copyright text and made it more friendly and compact.
I did also start work on a “Free Mod” mode for multiplayer, allowing each player to choose their own mods. There’s still quite a bit of implementation and testing to get this working perfectly, but expect it to hit the test build in the next couple of days!
Touching back on the start of this post, I am now in Japan for the near future. Give me a yell if you are in Japan (or planning to be over the next few months) and we may be able to catch up for a coffee, or game of jubeat ;).
As a side note, I was able to write this entry completely on my iPhone using the newly released Textastic code editor. Pretty awesome stuff.
Just a quick note that i will be skipping this week’s “week in osu!” due to being busy in real life and working mostly on the CtB hyperdash fixes with the time I had. Hopefully the post I made on hyperdash will make up for it :).
Over the last week, I have been working on fixing long-standing issues with the “Hyperdash” algorithm in Catch the Beat mode. Hyperdash is a gameplay tweak that was added to CtB in order to ensure that all patterns are catchable – no matter how far apart they are. It does this by makring certain fruit (objects) which would previously be impossible to catch as “Hyper” fruit. On catching these fruit, your catcher speeds up to the precise speed required to reach the next fruit in time.
This has always worked in roughly 99% of cases. Today I will detail the two main issues which caused the flaws in the remaining cases. For brevity, I have left out some other more obvious cases which were fixed earlier on last month.
The catcher needs to catch three (or more) fruit in a line, where all but the last are hyperfruit. In the initial algorithm, the current X position of the catcher was always considered to be centered on the destination fruit by the time required, but because hyperdash would disengage earlier than this, catching all fruit were not possible.
The first fruit is caught as expected.
The second fruit is caught, but the catcher is dashing in overtime to get there. The green silhouette is where the algorithm expected the catcher to be.
Because the algorithm expected the catcher to be further than it actually was, the third fruit is not catchable. Hyperdash was required earlier on in this pattern.
Fixing this issues involves keeping track of:
If the catcher has been moving in the same direction over more than two fruit, the excess required movement (if any) is added to the required distance, allowing hyperfruit spawning where required.
Hyperdash disengage times were also made more precise to ensure reaching the perfect position to catch the fruit.
The most edge case meant that some fruit may be possible or impossible to catch only by luck and frame limiter settings. Internal calculations where done on a sub-frame level, meaning:
possible_movement_distance = (object2.startTime - object1.endTime) * velocity;
require_movement_distance = abs(object2.x - object1.x);
if (possible_movement_distance < required_movement_distance)
hyperdash(object1);
The problem here is that the possible_movement_distance was calculated based on the number of milliseconds separating the two objects. As osu! internally runs at 60fps in the worst-case scenario, possible_movement_distance may have been less or more than the actually available movement, due to where calculation frames end up falling (which is completely uncontrollable and random).
Correcting this is a matter of rounding available time downwards to the nearest frame + 0.5, which ensures that in worse-case scenario the time calculation will be 0.5 frames more lenient than required (ie. the objects will still be catchable):
frame_time = 1000/60; //60fps (16.66667ms)
actual_frames_available = floor((object2.startTime - object1.endTime) / frame_time) + 0.5;
possible_movement_distance = actual_frames_avilable * frame_time * velocity;
When the game’s frame limiter was set to 60fps, frames were guaranteed to be a full 16.6ms, rounding calculations upwards and allowing slightly more movement. They would therefore match the old algorithm and be catchable without hyperdash.
When the game’s frame limiter was set to 120fps or unlimited, frames could be any length, causing higher precision in calculations, and introducing the possibility of sub-frame errors as seen above. While the fruit should technically still be catchable, rounding errors meant that they were not catchable around 50% of the time.
I spent the last week stumbling around trying to keep up with both real life as well as osu! commitments. I don’t have much to show this week so instead I will go a bit more in detail than usual on what I did get up to.
I got really annoyed at cross-compatibility of symlinks which are committed to git, so took a day to get rid of every symlink (more effort than you’d think!). The real issue only shows itself when you are using a git repository shared across multiple PCs using Dropbox.
Most symlinks were in place for large data stores which may be mounted across disks or PCs (using NFS). Removing them meant replacing with nginx location/aliases where possible, or absolute paths in other cases.
For example: ./repo/www/mp3/ → /data/large/mp3/
In this example, we see ./repo/www/mp3 inside a git repo pointing to an absolute path which may be a local folder or mounted hard disk. In the case of storing these in git, there is no initial issue – symlinks are a file with a special link attribute set and contain the path to their destination.
After checking out the repository on a unix based system, symlinks will correctly point to their absolute destination, so if you have a folder available at /data/large/mp3/ it will work instantly after deployment via git. This works great for automatic server-side deployment without having per-system configuration containing locations of certain sets of files.
I store all my git projects inside Dropbox, which means I can use multiple PCs simultaneously without a tiresome commit->push->pull to get both PCs in sync. When symlinks are synchronised across to windows PCs via git, windows is unable to keep the link attribute intact (it contains no such attribute), and the links become non-linked text/binary files. This causes a conflict between the two dropbox connected PCs and results in really weird scenarios which generally break git until you pause one PC’s Dropbox syncing and reset any file changes.
Long story short, I would highly recommend avoiding symlinks inside git unless completely required :).
Database was getting big, so I took a day to prune unnecessary data and do a clean-up of indices. I aim to do this about once a month (basically as soon as things start to fill up) to both keep the overall filesize within acceptable limits and keep the database running in an optimal state.
This generally involves:
OPTIMIZE TABLES n; command, and defragments remaining data in the process.End result was a 10% reduction in database size!
Ephemeral started widescreen support for the editor, and I finished it. Feels pretty nice to use! Also if you run osu! fullscreen, the editor will always default to borderless fullscreen now, meaning an almost imperceptibly smooth flow to the editor.
A lot of time spent trying to get Catch the Beat “Hyperdash” to work correctly for all patterns. Previously there have been issues in cases where objects next to each other should be hyperdahsed, but aren’t due to sub-frame calculation errors. Still haven’t finished work on this, and it is holding up the next public build, so is my highest priority at the moment. You can read about the specifics of fixing hyperdash in my separate blog entry.
I hope everyone has an enjoyable Christmas break. I took a day or two off, but spent the rest of the week focussing on some scalability issues creeping up on osu! – the main one being the initialisation process when first connecting to bancho.
Using the concept of caching user information locally, I reduced the amount of data from server→client by 90%. osu! can now play in solo mode without ever receiving more than userIds of all online players. When users need to be displayed or interacted with (chat / multiplayer / online users display) this information is requested on-demand. I am still experimenting with the best balance between local cache size and expiry times, but you should already be able to see the improvements when first starting osu! and connecting to bancho. THis should help out those with slow internet connections quite a lot.
I reworked much of bancho’s internal to treat all clients (osu! / irc) evenly. Previously there were a lot of special cases to handle multiple logins from the same account, including |osu suffixes on all osu! client’s username, and the inability to have an IRC client share a username with an osu! client in many cases.
These changes not only allowed dropping that ugly username suffix, but also means that a single user can have multiple IRC clients all running under a single unified username. In the process, I also removed the ability to change IRC usernames, so it should be a lot easier to identify people too (even though you could only add a suffix previously). The code for handling user connection is also a lot cleaner, and will allow for special cases like tournament spectating without having one osu! account per running client.
I fixed most of the remaining widescreen compatibility issues, and also went through a lot of hard crash bug reports and fixed them. osu! should be a whole lot more stable after this week’s updates. In addition, I changed the way crashes are handled so a lot of errors (>80%) that would previous hard-crash osu! are now handled (and reported to me) without the game dying completely.
I made some fixes to input handling which allow prioritising certain groups of game elements on a situational basis. The main reason for doing this was to allow universal volume adjustment – now available either using up/down arrows or mouse wheel, and in dialogs where it wasn’t previously by holding ctrl+shift while using the controls – but also allows for a lot finer control of input handling, especially in cases where there are many scrollable areas, buttons and overlapping layers on screen at once. Quite large internal changes, but quite invisible to the end-user.
I have started work on fixing long-standing Catch the Beat hyperdash issues, where fruit would occasionally be impossible to catch due to sub-frame calculation errors. It is going to take a bit more time to get things finely tuned without affecting the difficulty of existing maps, but should be ready for the next public build, if not the one after.
Let me take a moment to note that I have recently been streaming dev sessions on twitch.tv. Make sure to follow me if you’re interested in coming and watching, asking questions or whatever. It’s a lot of fun, and I play some epic music playlists to boot :).
Spent this last week finishing up work on widescreen background support during play mode, and took the opportunity to improve the transition between song select and play mode, along with the loading process.
All loading is now done asynchronously on a separate thread. This means there is no visual pause when loading beatmaps. While this doesn’t mean much for most people with fast hardware, it was a prerequisite to the changes I wanted to make to the transition into play mode.
For the time being, I added a fixed one second delay to the loading process to make the UX how I envisaged it working (especially important for fast PCs). I have further plans on what will happen during this loading process to make it more than just loading, but that will come in a future update.
Fun Spoiler has been renamed to Visual Settings, and buffed a bit. It is now possible to dim background images to black after playing maps at least once. The new dialog is also featured in a much more prominent place, hopefully meaning more people will see and make use of it.
Widescreen support for play mode, with aspect fill now used for all backgrounds and videos. Automatically detects when storyboards are present and falls back to 4:3 display, to ensure alignment is not broken as a result. There are still a few cases where artifacts can be seen here, suck as storyboards which used to have hidden elements due to letter-boxing now being visible. This will be fixed and improved going forward, along with the option to make widescreen storyboards.
As a side note, you can now start using widescreen backgrounds and video and they should be recognised appropriately.
I made some fixes to opengl display – especially to the initialisation process. This means that opengl should load faster than ever and no longer flicker on load. I also looked into getting context sharing over multiple threads, to bring opengl up to the same level of threaded performance is direct x. One more step forward towards dropping direct x support completely – part of my master plan of ridding osu! of XNA and supporting more platforms more natively than ever before.
Added the ability to set custom refresh rates (for fullscreen mode) and custom fps targets for frame limiter mode. Should be useful for people playing osu! on 120hz monitors, or those who are after a frame-rate limit somewhere between unlimited and 120fps.
Best of osu! 2012 voting is now open! Make sure to check it out and vote for the beatmaps you enjoyed the most over the last year.
I have quite a bit of stuff to organise over the coming weeks, so updates may be a bit slower than usual. I do have plans for a small osu!stream update, including a new map pack, so keep an eye out if that kind of thing interests you :).
Running a bit late with the post this week, due to an all-weekend LAN event which involved updating and taking along my two osu!arcade units! Currently playing catch-up with both sleep and work.
I again spent some time optimising the tourney spectator client during the weekend’s OWC matches. Mostly just some minor tweaks to ensure smoothness and reduce buffering.
Made some UI changes to the updater – mainly that the “use test build” checkbox has been removed. Some people seem to have been accidentally clicking this without understanding what it means, and locking themselves into the test build. Also looking at changing the update process a bit. Not too happy with how the updater is such a context switch out of the game.
Switched CDN/static content over to https with SPDY support. As most major browsers support SPDY, there is no need to distribute content over multiple domains to achieve parallel throughput, so I also moved all javascript resources locally. Based on my own performance tests, the difference was not discernible, but I have had positive feedback from countries such as China, where this kind of thing matters.
Improvements to lobby layout, making better use of available screen space. The filters toggle is no longer available, with filters always being shown instead. I have plans for further changes here, but still brewing in my head. The users list is no longer displayed, saving on a bit of network traffic as well.
I rewrote the in-game chat link parsing algorithm. This was a pretty big endeavour, replacing some at-times-hacky character counting for link styles like [http://test.com test words [may contain brackets]] with single regex match-and-replaces:
\[([a-z]+://[^ ]+) ([^\[\]]*(((?<open>\[)[^\[\]]*)+((?<close-open>\])[^\[\]]*)+)*(?(open)(?!)))\]
Explaining this would be require a separate post, I think. Let’s just say that chat links now support line breaks, cases where multiple links are on the same line, and gracefully display erroneous links. They also look a bit nicer!
Messaages in chat by silenced users is now removed from in-game chat history. This ensures that bad behaviour – especially linking inappropriate content – has minimal effect on other users. Took this idea from twitch/justin.tv’s book.
Hosted osu!arcade units at RFLAN yet again. This included adding experimental support for conversion of osu! maps to osu!stream, which is working pretty well! I spent about half the LAN watching people play and modifying the game to provide a better experience based on this feedback.
Make sure to check out this highlight of an osu! map being played with a straw, or dive in and watch some of the recorded stream of the 24 hour event!
I also released an osu!stream update on the App Store with iPhone 5 support.
I’ve got some interesting stuff planned for osu! updates in the near future, so stay tuned :).
This week I set aside a good block of time to replace the registration and activation process for osu!. The main reasoning behind focusing on improvement in this area is the number of emails I receive each week from people who have issues with either the captcha or email verification/confirmation, requiring manual intervention.
Registration process was completely rewritten to be as simple as possible. Rather than hinging on standard verification methods, new users are now only asked for their username, email and password (and only once for each). Once these are validated, they are asked to login to osu!.
The login to osu! not only double-checks they entered the password they think they did, but it also means they are instantly logged in without any annoying email checking getting in the way.
The main disadvantage to this change is that anyone who cannot access an osu! client will not be able to register. This means that (for the moment), osu!droid or osu! iPhone users will not be able to participate in the forums. I may provide a solution for these people in the coming week(s).
For a limited time, you can test the new registration process for yourself on the test server. Note that you will need to run the test build of osu! to complete activation. Accounts created here will be deleted around every three hours, so don’t worry about making more than one (or entering valid details).
When creating new accounts, users are now warned if it is likely they are about to (unknowingly) create a multi-account.
The download page got a much needed redesign. I plan on doing some more work here, but it is already feeling a lot nicer.
Next up is fixing the email change, password change and forgotten details processes to bring them in line with the quality and simplicity of the new system. I hope to complete these next week.
Tournament spectator client has some upgrades! It now synchronises playback between all clients, which looks pretty damn cool in practice. I can’t wait to see what people think during the OWC matches this weekend.
The spectator setup process is now completely automated, meaning streamers no longer need to track player or song changes. On providing a multiplayer match ID, osu! will handle the rest. Automation Ho!
Previously mentioned bancho protocol optimisations have gone live and have reduced overall outwards traffic by around 5-10%. This is quite a substantial improvement – especially since this saving occurs during game client connections, reducing the delay before users can enter multiplayer etc. by around 20%. I still have a few more improvements up my sleeve for further down the track :).
I changed the file structure of avatar storage to allow for easier access from multiple sources. File extensions are no longer provided, and as such file types are determined during client-side display logic. As a result, there are no longer conflicts when users upload multiple formats of avatars over a period of time, and the logic for displaying avatars is super-simple. As an interesting tidbit, osu! has over 5.6gb worth of 128x128 avatars in its store!
Going to be another busy weekend with OWC matches throughout Saturday and Sunday. Make sure to tune in and witness the improvements to the streaming setup!
With the OWC tournament ongoing, I have been helping out in that area and further improving broadcasting tools. As a result, not much visible progress was made this week.
Multiplayer history pages on the osu! website now update in real-time! I really wanted to provide an up-to-date version of scores without forcing people to mash the refresh key, and am happy with the result. Updates propagate within 5 minutes of in-game events. I also re-designed these pages to be more readable, and use the new beatmap info panes.
I have made further improvements to the tourney spectator client, removing file access conflicts when downloading new beatmaps and/or processing the beatmap database. Each of the clients now communicates with each other to schedule updates, with the first client being the only one to perform the grunt of the work. Country labels were also added, so streaming plugins are no longer required for these details.
When making some web changes, I found some code which was used for user rating of maps that had not yet been updated to using the latest database API, so I spent some time fixing this up. In the process I removed a good 0.5gb of redundant data and improved the efficiency of beatmap ratings by a whole lot!
A lot of new localisable fields were added including newer options screen text (by woc) and main menu tips (by me). I made further improvements to the usability of the localisation spreadsheet, as well. There are at least three new languages this week which went from non-existent to 100% completion. Let’s say I am very impressed with this outcome – although I’m not as impressed that the Japanese translation is completely stagnant at lower completion than the existing v1 translation (any takers?!).
I have started automatically reporting beatmap corruption to the server so I can figure any remaining causes of this and iron things out.
I have begun micro-optimisations on the bancho network stack to reduce the size of user presence packets and hopefully speed up the initial osu! server connection – which can be seen struggling when there are 8,000 users. It is currently sending about 250kb of data, and so far I have reduced this by 35%, and hope to gain further reductions before making this live.
I received my t-shirts and hoodie order from the osu!market, and am thoroughly pleased with the quality! We have also recently added some mugs to the store with the same designs, so go check them out if that is your kind of thing :).
As a reminder, we have been streaming the osu! World Cup via twitch, and hitting around 3.5k viewers at peak, so make sure to check out some of the matches (the schedule is listed below the stream, and you can also access recording as well in the “Videos” tab).
Another relatively busy week on the server maintenance side of things, switching out the old bancho server with the new one. There also seems to be a sudden surge in tournaments being organised, which triggered a temporary change in priorities!
Bancho is now running on the new server. The switch was made at low-peak on the 19th, and there was no downtime experienced. This did require an osu! client update, which was rolled out over a few hours beforehand. Non-updated clients will not be able to connect, but as always, are automatically updated.
The new server is provisioned for an extra 66% bandwidth (allowance, not speed), and should be able to handle increasing traffic for well into next year.
I shifted server monitoring to a hosted service (ScoutApp). This was primarily done to alleviate the risk of failure (and maintenance) involved in running monitoring myself, which was previously done using Cacti.
ScoutApp was my choice for monitoring because it offers reasonable pricing and easy setup. I am still running my monitoring using their one month trial, but am happy enough to continue using it into the future.
The ability to log many stats and create dynamic graphs at any point in the future makes ScoutApp super powerful. It also has an open plugin architecture, which allowed me to graph .NET metrics even though they don’t actually support Windows Server as an OS.
The screenshots page now shows a selection of relatively-popular user uploads (those things that come from hitting Shift-F12 in-game).
I implemented a new spectator mode for tournament use, which automates the process of setting up an array of spectator clients to live-stream tournaments. This will be used for streaming the upcoming osu! World Cup #3 (starting in just over six hours, running for a month or so) and a smaller Japanese osu!mania tournament running this weekend (Saturday UTC11:00~).
I have plans to further improve this functionality and make it semi-public, so anyone wishing to use it can apply for temporary access. It is currently available via private applications. If interested, please make sure your tournament is well set-up, has a thread and reliable staff and is fully scheduled before contacting me via PM for access.
While investigating performance issues (latency occurring on the last hitcircle of a beatmap), I accidentally found other areas of osu! I could optimise. This ended in a rewrite of texture handling which means that when running in Direct X mode, osu! no longer needs to store a copy of textures in RAM when they are already sent to video memory. This resulted in ~33% decrease in memory consumption across the board – and even more in some cases.
The reason textures were stored in RAM was due to XNA allowing developers to be more lazy in how they choose to load textures, meaning if a graphics context is lost (window changes to another monitor, resolution is changed, full-screen toggle etc.) there is no extra code complexity to reload textures. osu! already handles this correctly and having XNA still managing textures was completely useless. I was very happy to find this one, as it was a quick win which will help users with low-spec machines across the board!
I found and fixed a memory leak occurring when loading custom hit-sounds. After loading any sample, the sample was being freed using an incorrect call, leaving some resources hanging on to invalid memory references. This may have caused crashes or excessive memory usage after several hours of playing maps using custom samples.
Localisations for osu! have been updated to use the new Google Sheet localisation method. This should allow for very speedy changes and additions to localisations. A total of 26 languages are now available for selection in osu!.
Every time I build a new public release of osu!, new translations are automatically pulled from the shared spreadsheet. I do take the time to look over changes and make sure there have not been any malicious modifications/additions (since it is publicly editable) before pushing them out to users. That said, localisations are not tied to individual osu! builds, and will be updated automatically when available.
The code to integrate reading from google docs was handily borrowed form osu!stream, where it has proven to be a very efficient method of localising in-game text. I do plan on adding more localisable text in the near future, including eventual editor support.
That’s all for this week! Let me know if you want less or more detail; if you want to know about different aspects I have failed to cover; even just that you enjoyed this post :).
I’ve decided to start this forum as a place to discuss what has happened over the last week in osu!. I’m taking this idea from the Webkit/Chromium developers who use a similar approach to sum up changes at a higher level than the raw changelog, which may not reflect where time has been spent over the week. A lot goes on behind the scenes of osu! and I’d love to share more of this with you guys :).
Here goes!
Recently I have been heavily involved in Apple platform development — specifically of the mobile variety. This has involved launching and maintaining a prominent e-book reader (with a user base of 500k+) alongside my personal projects’ apps (puush and osu!stream). While I generally try to see the best of any situation, developing for iOS has fueled stress and anger that I am not used to, and even made me consider at times that changing profession wouldn’t be such a bad thing.
And don’t get me wrong, I love coding. I live for code. So something must be drastically out of balance here, right? While some of the issues I have are specific to my own workflow and methodology, I think most of you will be able to agree on some level that things could definitely be greener in the Apple ecosystem. I’m going to touch on a few specific areas of iOS app development over the last couple of years which have hampered my productivity. Hopefully it will serve as a light warning to those looking at entering this arena.
Apple always has the last say
Since releasing our e-book reader, we have had to remove over half the functionality in order to keep it live on the store. After Apple decided to enter the e-book market, it was initially unclear as to how we would be affected; I knew there would be a drop in sales as Apple leverages their App Store and OS strongholds to promote their new app and store, but they did not stop at just that.
In a manner that I would personally say is many times worse than the age-old Microsoft anti-trust case, Apple gradually changed their policies to lock competitor e-book providers out of their system. It is obvious that they were not happy having customers using their devices to purchase books via other providers (even though all fulfillment was done via an external web link), and gave us a deadline to make the transition to In-App Purchases. This would mean giving a 30% revenue margin to Apple.
While reluctant to take this approach, we did end up attempting to embrace Apple’s restrictions. Sadly, the publishing industry is a very complex one, and payments and pricing could simply not be moulded into Apple’s tier-based pricing system. This would not allow for regional pricing, state-based tax, 100k+ product listings and rapid price changes, to name a few problematic points. Of course, Apple would have understood this from the beginning — having to abide to the same contractual terms with publishers themselves when distributing via iBooks — and pushed the deadline forward further while they planned the road forward for their competitors.
Then, along with Amazon and other key e-book providers, we were given an ultimatum: remove the store completely from the app. “Okay”, we said, and replaced the in-app store with a link to our website. This was not enough to please apple; within a day our app submission was rejected, and what followed were a few months of back-and-forward build rejections. The end result wasthe removal of every UIWebView from our application, along with every external link. Yes, we could no longer provide even a web link taking customers to our company’s own web site (not even a “forgotten password” link!).
The end result is a thin-client reader which can do not a lot more than read and sync books purchased outside the iOS environment. Potential customers cannot make a purchase unless they manually make the connection via our app and our website.
Too late to apologise
When I initially made osu!stream available, I ran into some licensing complications with the music I was using, resulting in the need to quickly pull the app (of my own accord) and replace the songs. A month later, I submitted an update to the app, which amongst many other improvements, mapped the previous song packs to newer ones. This was a way of ensuring users which bought the initial song packs didn’t feel cheated after they were made inaccessible.
To reach the maximum amount of users possible with instructions on how to retrieve the new content (and providing a brief explanation of the events that took place), I wrote up release notes which had a paragraph dedicated to the licensing issues. In this paragraph, I apologised for what happened. This update got rejected on basis that release notes were “not related to the changes in the app”.
When Apple reject an update, they generally do not give any specific instructions, instead only quoting the relevant sections from the Terms of Service passage and leaving the rest to you to figure out. I had a hunch that the word “sorry” triggered alarm bells, so changed only that sentence; accordingly, it was approved one day later (n.b. they did actually check each localisation, so it took two shots!).
While this might not seem like a huge deal, it is just another example of the kind of randomly exerted control they have over the approval process.
As agile as a snail
One of the more apparent downsides of iOS development is the compulsory approval process associated with deploying an app to the App Store. Depending on seasonal demand, this approval process can take anywhere between 6 hours and 14 days (based on my experiences to date). The lower end of the scale is generally only seen when it is in Apple’s interest to get it live quickly.
I usually work with a very rapid release cycle, usually aiming for daily releases — pushing features and fixes the moment they are mature enough. At times this means relying on the end-user to do the “testing”, but I find this to be the most productive release method. Keeping things fresh generally keeps people happy and coming back for more. Maybe I am designing for myself here, but if I was a user, I would enjoy seeing my feedback rapidly incorporated in a product.
Being limited by external forces to an average of weekly releases — with no rapid user feedback in the interim — makes development very staggered and unproductive for me. This is likely amplified due to the way I work, so I would be interested to hear other people’s experiences with the imposed time-between-release limitation.
You might say “but you can deploy to testers as often as you want via a rapid deployment process (testflight is awesome)!”, to which I would answer that I am definitely aware of this and actively deploying regular builds using these methods. The main problem here is that each developer account is limited to 100 registered devices for device provisioning, and while this may sound like a large number, I found that a month or two after accepting testers over 80% of them were inactive. You are only given one opportunity each year to remove a number of devices (at the anniversary of your account registration), which is both cumbersome and sub-optimal.
There are also many documented examples of extreme cases where apps get stuck in a pending state with absolutely no response from Apple. 2Do is one current example, who seem to be stepping in dangerous territory due to their iCloud’s CalDAV sync. I have had some personal experience in trying to get a response from Apple’s chain-of-command via phone and email and can attest that it is not so easy. Lack of control over your own product’s release cycle — and a communication blackout from those in control — is indeed quite a scary concept.
Think Different, or Don’t Think at all?
The final straw (which actually triggered the writing of this article) was having my app rejected today for storing downloadable content in the application’s “Documents” folder. Since the introduction of iCloud, the guidelines have been changed to only allow “user-created” documents to be stored in this specific folder. This wouldn’t be such a bad thing if it wasn’t the only non-volatile storage area available to developers.
In short, this means that if myself and other developers in a similar predicament are to abide with the new terms, iOS can choose to delete all downloaded content in any app it sees fit to free up memory. Users are not given the option to disable this behaviour — let alone warned about it when it happens. This is a support nightmare for app developers, who users are going to obliviously blame for the loss of their files.
The new “cleaning” behaviour (more details in this article) is going to surprise many users over the coming months as their files disappear at iOS’ whim. In osu!’s case, this means all purchased songs would disappear, requiring special support at an application level to track when files have disappeared, alert users and trigger a fresh (and still quite awkward) in-app purchase process to re-retrieve the content. With a bit of luck, the user will have an internet connection available at this point, but what happens if they are in the middle of a long flight, or in an area without coverage?
I have been following this behavioural issue since it was discovered earlier this month, but didn’t think I would be affected — after all I am only storing around 10mb (at most) to the Documents folder. It would seem that for whatever reason, Apple is not only actively enforcing this new clause, but cracking down on it hard.
Waiting a week for approval to only be rejected is quite a bad feeling (especially when you have other coinciding release details planned), but when the rejection is due to this kind of issue it really makes you wonder if Apple know what they are doing. And makes me ask myself whether I really want to continue developing for a platform this unstable.
—
Hopefully this didn’t come out as too much of a rant — I do always try to look at things from a neutral perspective. I won’t be stopping iOS development, but may take a few days off to regain some motivation :). I am not changing my app’s behaviour to meet the new clause requirements and have instead filed an appeal with Apple. I hope there is enough velocity from my appeal, along with other affected developers/users’ voices to change the way things work.
So I have been using parallels for a while now, and it is turning out to be a very good experience. I would like to write a longer post on the ups and downs of the application itself, but this post is a solution to a long-standing gripe I’ve had with the product line — the dock icon.
“Paused? No, that’s our logo. You can’t remove it, either.”
I’m sure you can agree that not only are the Parallels bars ugly, but they also communicate a state which is actually incorrect. The resemblance of a pause symbol is undeniable. Having this icon in my dock is an eyesore and a distraction due to the fire-truck red colour.
Fixing it is unfortunately not so straight-forward, and while many people have attempted to cry for help, those behind parallels have refused to listen. For three years.
I offer a slightly hacky solution that should work for all versions 4.0+ and into the forseeable future. If you use this solution and your PC blows up, I am totally not responsible.
Open Terminal (Applications > Utilities > Terminal.app).
Type the following command (as a single line):
LANG=C sudo sed -i.bak s/Parallels_Desktop_Overlay_128/Parallels_Desktop_Overlay_000/g /Applications/Parallels\ Desktop.app/Contents/MacOS/prl_client_app
Enter your root/administrator password and wait for the command to complete running.
Congratulations; your Parallels icon should be a touch less ugly now. The above command creates a backup of the file that was changed, so if you would ever like to undo the changes, simply run (as a single line):
sudo mv /Applications/Parallels\ Desktop.app/Contents/MacOS/prl_client_app.bak /Applications/Parallels\ Desktop.app/Contents/MacOS/prl_client_app
I am still looking into whether a completely custom icon can be used (so parallels doesn’t switch to the windows-desktop-monitor icon), but haven’t been able to find the source of that one yet.