Version 0.9.11 is out.
Version 0.9.11: --------------- * General: * Work around bug in Wesnoth 1.11.17 that could cause units to be deleted from the game by units moving during cutscenes.
It’s almost as if moving Web hosts in October is going to become a yearly activity, like New Year celebrations, or birthdays.
shadowm.rewound.net turned into a testing ground for crazy experiments ranging from a seldom-updating blog, to a quotes database for an IRC channel. Giving the few interested visitors a general idea of who I am and what I do in my spare time was barely a goal at first. Fast-forward to Q4 2014 and just look at this mess.
But as everyone knows, nothing is eternal. Not even the sun. Okay, from a practical standpoint we can assume the sun is eternal. That’s beside the point.
The first chapter of “shadowm”— formerly known as “Shadowmaster’s Lair” — lasted much longer than it logically should have, and I thank grafix for that. Starting today, I am now hosted by AI0867, a close collaborator of mine in projects like Wesnoth-UMC-Dev, my campaign Invasion from the Unknown, and, well, Wesnoth itself. Hence the new hostname,
Around May this year, I went to great lengths with the Iris site redesign to ensure everything would work on any host provided the software dependencies are satisfied, and removed and/or optimized a load of cruft left from earlier iterations. In theory, things should perform more or less fine on the new host, although it is hard for me to gauge this as a user — for you see, I am stranded in high-latency mobile broadband land, making pretty much every website out there equally slow from my point of view. In any case, if there are any glaring inefficiencies, I’ll do my best to correct them over the course of the next few days.
To sum it up, most people (i.e. the handful of followers I have) should not notice a difference besides the new hostname in URLs. Old links will continue to work for approximately a year through the magic of HTTP redirection, though you should probably update them now just in case.
Anyway, that’s all I wanted to say. Now back to our regularly scheduled programming.
Version 0.9.10 is out.
Due to circumstances, it has been quite a while, and honestly I lost track of what this release was supposed to have besides a thing that requires another thing from another campaign that has not been completed or released yet. So let’s talk about what AtS 0.9.10 actually has.
Firstly, the minimum Wesnoth version requirement now is 1.11.11.
That’s right. Previous versions (including 1.10.x) are no longer supported. Ever since I moved to 1.11.x following the release of AtS 0.9.0, maintaining support for previous versions (including buggy development releases) required a series of unwieldy kludges that made the code uglier and harder to maintain and were, for the most part, untested beyond the classic “does it compile?” test. With all those bits gone, it will be easier to improve and optimize some aspects of the campaign, as well as work on the thing that requires the other thing I alluded to above.
For now, the first one such aspect I have worked on is converting several units to the 1.11.x animation WML syntax. Although some other people seem to prefer the new syntax over everything, I have chosen a more pragmatic approach for this campaign, so the set of units that I’ve converted in this release is rather limited. Hopefully more will follow soon, but I’m certain that there are a few for which the change hurts code readability. Plus, since most of my units are headbutters, the code size gains are marginal in the average case.
Finally, somebody reported to me of an issue with the player’s recall list and gold being discarded during certain key scenario transitions. It turns out this resulted from a change in Wesnoth 1.11.13 purportedly intended as a bug fix for MP campaigns. I was aware of the change and its implications at the time 1.11.13 was released, but I wrongly assumed AtS would not be impacted because I failed to take a tiny detail into account. Exactly three months later, I realized the sheer gravity of my mistake — but fortunately, it seems nobody else played AtS on 1.11.13+ in the meantime. (Thanks to RainerT for the report. This would have gone unnoticed for who knows how many more
years months otherwise.)
As you can see, there is not a lot to talk about in this release other than the version requirement change. Since it’ll be a while before Wesnoth 1.12.0 is released, and AtS remains largely the same as it was the last time I posted in this topic, I believe stable version purists won’t be missing out on anything for now — at least not until the thing is done.
Also due to circumstances, this release is largely untested, so I would not be surprised if I accidentally broke a thing or two since 0.9.9.
The complete changelog for this version follows:
Version 0.9.10: --------------- * General: * Raised minimum game version requirement to 1.11.11. All existing compatibility code for previous versions has been removed. * Graphics: * New or updated unit graphics: Sprite, Fire Faerie, Forest Spirit, Dryad, Demon Shapeshifter. * Scenarios: * Added an option to certain scenarios to ensure Wesnoth does not discard the player's gold and recall list under certain circumstances due to a behavior change in version 1.11.13 and later. Affected scenarios: * E1S9.3 - The Triad, part 3 * E1S11 - Return to Wesmere, part 2 * E2S0 - Transience * E2S11 - A Final Confrontation * E3S0 - Opening (Within) * E3S6 - Divergence * E3S8B - Destiny, part 1 * E3S11 - After the Storm * Units: * Balancing changes: * Changed Leech's alignment from 'lawful' to 'neutral'. * Decreased Leech's HP from 62 to 42. * Decreased Leech's melee damage from 11-2 to 9-2. * Decreased Leech's unit level from 3 to 1. * Converted to the simplified 1.12 animation syntax: * Dusk Faerie, Night Nymph, Nightshade Fire * Sylvan Warden * Sprite, Fire Faerie, Dryad, Forest Spirit * Elvish Wayfarer * Faerie Avatar * Demoness Hellbent Tide * Verlissh Control Spire * User interface: * Cutscene themes now use the 1.11.10 [theme] id attribute on 1.11.10 and later.
Those who have followed me these last five years or so are probably aware that I designed and coded this site’s layout on my own in an effort to learn the basics of Web design. Thus,
shadowm.rewound.net has been redesigned no less than eight times since its inception.
Dorset6, the last iteration prior to this day, was first deployed on May 11th, 2011. Since then, it has only received minor incremental updates addressing bugs and minor practical concerns that cropped up later, since I found the overall design to be pleasing to the eyes — at least until I got a new screen where the lighter blue colors turn out to be purpler than intended, oops!
The last maintenance update to Dorset6 was Delta 24, deployed on April 26 2014.
The underlying template engine, Poison Ivy, has not seen any architectural changes all these years since it was first deployed in 2010. In the meantime, however, I have learned more about API design from my experience with various other projects. It can be very disheartening to look back on past practices and realize you were doing things completely wrong in production software!
During this time I have also heard all the hype surrounding HTML5 and CSS3, and got to experience the results firsthand as large websites started making use of the new technologies. On top of that, support for Internet Explorer 6 is finally dying off, and Windows XP just reached its definitive end of life last month; software and Web developers everywhere rejoice as they no longer have to give support to grossly obsolete products from the early 2000s.
I have actually made use of CSS3 and forward features in
shadowm.rewound.net since Dorset6, but right after deployment I decided to aim for leveraging more of their potential in the next iteration, Dorset 7. I promptly begun throwing around style and layout ideas. However...
WML is the primary source code language for all add-on and mainline content for Wesnoth. Creators like me write large amounts of WML to instill life into their user-made projects. Our wiki has a rich reference section for the various dialects of WML understood by the game in different contexts. Tools like wmllint exist to make sanity-checking and porting tasks easier for
the few people who can tolerate its obtuseness us. There is a whole forum section in Wesnoth.org dedicated to assisting people with their WML endeavors.
With so many resources available, you’d think WML is the most fun thing to work with, right?
One of the most annoying aspects of add-on development is that whenever something goes horribly wrong with the WML document’s well-formedness, you may get a barrage of file paths thrown at you in a superdense text-wall format, particularly so if the error site has been subject to file or macro substitutions, which is the case with pretty much every add-on that has more files than its top-level
_main.cfg. Trying to find the relevant portions of the substitution trace to debug the issue can be frustrating at best.
When more than one add-on fails to load, though, the results can be even messier.
The report is displayed on the screen by calling a generic GUI function that creates a new instance of a generic message dialog with the specified contents, which are also printed in stderr. Since the only formatting option available for the dialog is to rearrange the text with a combination of whitespace and newlines, the same thing winds up in
stderr in the same format:
20140216 10:50:12 error config: error reading usermade add-on '/home/shadowm/.wesnoth-1.11/data/add-ons/After_the_Storm/_main.cfg' 20140216 10:50:12 error general: The following add-on had errors and could not be loaded: /home/shadowm/.wesnoth-1.11/data/add-ons/After_the_Storm/_main.cfg ERROR DETAILS: Found invalid closing tag [/unit] for tag [side] (opened at ~add-ons/After_the_Storm/episode3/scenarios/13_Epilogue.cfg:98 included from ~add-ons/After_the_Storm/base-loader.cfg:20 included from ~add-ons/After_the_Storm/_main.cfg:174 included from ~add-ons/After_the_Storm/base-loader.cfg:108 included from ~add-ons/After_the_Storm/_main.cfg:174 included from ~add-ons/After_the_Storm/episode3/scenarios/13_Epilogue.cfg:101 included from ~add-ons/After_the_Storm/base-loader.cfg:20 included from ~add-ons/After_the_Storm/_main.cfg:174 included from ~add-ons/After_the_Storm/base-loader.cfg:108 included from ~add-ons/After_the_Storm/_main.cfg:174), value ']' at ~add-ons/After_the_Storm/episode3/scenarios/13_Epilogue.cfg:94 included from ~add-ons/After_the_Storm/base-loader.cfg:20 included from ~add-ons/After_the_Storm/_main.cfg:174 included from ~add-ons/After_the_Storm/base-loader.cfg:108 included from ~add-ons/After_the_Storm/_main.cfg:174 included from ~add-ons/After_the_Storm/episode3/scenarios/13_Epilogue.cfg:112 included from ~add-ons/After_the_Storm/base-loader.cfg:20 included from ~add-ons/After_the_Storm/_main.cfg:174 included from ~add-ons/After_the_Storm/base-loader.cfg:108 included from ~add-ons/After_the_Storm/_main.cfg:174
This issue affects mainline and UMC developers alike, and yet, it has gone unsolved for a whole decade of development, even while other usability issues have been and continue to be addressed. Perhaps the very fact that the problem has existed for so long has silently perpetuated a misguided notion of impossibility — because, surely, if fixing the relevant code was that easy, somebody would have already done it? Or maybe, just maybe, everyone is afraid of altering code that lies deep within the core of Wesnoth’s WML processing pipeline; because it is an indisputable truth that the WML preprocessor and parser aren’t the most accessible modules for inexperienced contributors, and changing functionality that constitutes the circulatory system of the game engine is obviously a risky thing to do.
But wait! Wouldn’t it be possible to just sort of... make the error report presentation nicer without meddling too much with the implementation details?
While the basic proposal sounds simple and the results of my two-day-long coding effort made it into Wesnoth 1.11.10 (a.k.a. 1.12 beta 1), it turns out that there are many more factors to take into account, and in the end I did have to mess with parser and preprocessor implementation details to a greater degree than I feel comfortable with.
Against what one would expect, there is only one exception object class in existence for parser errors, and it is extremely generic. Callers have no way to tell what just happened when WML parsing has failed, other than asking the exception object for an error message... which is both stored and delivered in plain string form. The error type, the error site, and the WML preprocessor substitutions that led to it; these are all provided to the caller in an atomic string that may contain portions translated to the user’s language beforehand. This means that we (the parser’s callers) can’t reliably scan the error message to choose the format in which we are going to present the error to the user. All we can do is grab that big-ass message string and throw it into a dialog.
The dialog used to display the report in versions up to and including 1.11.9 is part of the GUI2 framework, but the particular dialog class in use is a plain generic message box. Naturally, it is not optimized for presenting WML error messages to the user — or displaying overlong strings in general, really. It is for this reason that the first part of this little project of mine consisted of designing a new dialog class (
gui2::twml_error) for presenting this information to the player in a more useful fashion.
Because the Wesnoth config manager will try to load all add-ons at once, the first thing that the dialog needs to do is to display a list of add-ons that caused a parser or preprocessor error when attempting to load them. The 1.11.9 approach is to hastily concatenate a newline-delimited list of paths to the files Wesnoth attempted to load, which are nearly always the
_main.cfg files. This can be quite misleading in most cases, and newbies might try to look for the faulty code in the listed
_main.cfg since it is the very first thing displayed in the error message.
gui2::twml_error instead displays the user-friendly name (1) of the add-on that was stored in a special file (
_info.cfg) when it was first downloaded from the server — failing that, it makes its best guess by replacing underscores in the directory name with blanks. If multiple add-ons failed to load, they are shown in a nice bullet list.
Where there used to be more plain dialog text, there is now a pretty scrollable box with the report contents (2). The report itself contains the list of errors found when loading the affected add-ons, but it is no longer all thrown at your face at once — the error log for each add-on is followed by an additional empty line, so telling them apart should be far easier than before. Finally, right above the report there is a button (3) to copy it to clipboard so players don’t need to take a screenshot of the whole thing to give to the add-on maintainer(s).
To be perfectly honest, the only reason that copy-to-clipboard button even exists is that I find error message screenshots irritatingly cumbersome to work with at both ends. It doesn’t help that I’m usually on slow Internet connections where downloading a screenshot of a Wesnoth instance surrounded by some fancy window decoration and taskbar and (occasionally) desktop wallpaper can be excruciatingly painful. Furthermore, as a forum administrator who has to keep an eye on how people use attachments, this preposterous waste of disk space can be quite annoying.
But replacing the error report dialog and the presentation of reports as a whole is not enough to improve the WML coder’s experience.
gui2::twml_error would still display the text wall-like errors if it were asked to do so by the config manager. Even though I hinted at the essentials above, the muddy implementation details screened by
gui2::twml_error would still fill up a whole blog post on their own. Thus, I am going to leave that story for another time.
04:51:30 <shadowm> aeth: I like to think of PHP as a recreational drug.
04:52:03 <shadowm> Everything is easy and cool until the effects pass.
04:52:16 <shadowm> Then you realize you are making your life more miserable by relying on it.
04:52:26 <shadowm> But you keep coming back!
04:52:51 <shadowm> Kids, remember, just say NO to PHP!
― ##shadowm, 2014-02-04
Version 0.9.9 is out, just in time for the holidays!
Since I have tested and developed this campaign primarily on Wesnoth 1.11.x since the 0.9.0 release, this version promotes support for Wesnoth 1.11.8 to official status. Because most of the campaign makes use of 1.11.x-specific features (both from me and other mainline developers) when available, it is quite possible that I will entirely drop support for Wesnoth 1.10.x in a future release even before the first Wesnoth 1.12 beta arrives. I still intend to make sure certain additions and changes for episodes II and III land before the last AtS version supporting 1.10.x, if time permits.
This 0.9.9 release primarily deals with prose corrections and improvements, and various other ‘cosmetic’ changes. There are also various fixes for some instances of dysfunctional AI recruitment on Wesnoth 1.11.7 and 1.11.8 resulting from the
recruitment_save_gold aspect being enabled by default in those versions (but not 1.11.9 and later).
There isn’t much in terms of new graphics since the prose and code changes (plus some mainline stuff) have kept me far too busy to do much more than some doodles. On the other hand, this release contains various animation fixes and improvements to the Aragwaithi units ported from Era of Chaos. There are also a handful of balancing changes affecting both Aragwaith and non-Aragwaith units.
Also featuring in this release are a number of WML optimizations intended to reduce campaign load times — especially on 1.10.x, which has a slightly slower tokenizer implementation than 1.11.x. Since the affected bits of code have been completely rewritten in Lua, it is possible that I accidentally introduced new bugs in the process that I may have missed during my playthrough, extensive as it was.
There is also a new secret feature that is not mentioned in the changelog. What is it, you ask? Well, if I told you, that would ruin the surprise. Think of it as a Christmas present!
Finally, from this release onwards, After the Storm is no longer part of the Wesnoth-UMC-Dev Project, and will be hosted on GitHub instead. Ever since development of the campaign started, Wesnoth-UMC-Dev provided SVN repository hosting for both After the Storm and Invasion from the Unknown, but over time that has proved to be an inefficient solution due to technical and organizational concerns. Although the conversion process was not easy in the least, I believe that this move will make things easier for me in the long-term, since I had been using git-svn to work on IftU and AtS since late 2008 anyway.
Release tarballs will continue to be hosted by Wesnoth-UMC-Dev for the time being, until I decide to phase them out entirely in favor of GitHub’s Releases page. If anyone is using them because they cannot normally download AtS or AtS_Music from the wesnoth.org add-ons server, I’d appreciate it if you let me know so I can make a more informed decision in the future (or point you to add-ons.wesnoth.org, that works too). For the time being, the tarball compression format has changed from Bzip2 (
.tar.bz2) to xz (
Special thanks to vultraz and 8680 for their proofreading assistance, without which this release would be about 1% less awesome. Also thanks to the current and past Wesnoth-UMC-Dev admins, AI/AI0867 and Espreon, for their continued support all these years — IftU and AtS simply wouldn’t be the same without Wesnoth-UMC-Dev.
The complete changelog for this version follows:
Version 0.9.9: -------------- * General: * Removed Wesnoth development versions warning from the campaign menu entries as support for 1.11.8 and later is now mature. * New complete algorithm for calculating the relative direction between two hex grid locations, handling all six intrinsic facing directions instead of only SW and SE. * Updated Aragwaith faction from Era of Chaos 1.3.1+dev up to commit 9dedeba7cddc2a027745c9994a917fdcb78ed341. * Stripped optional whitespace from terrain map and mask files, decreasing uncompressed directory size by about 62%. * Graphics: * New or updated unit graphics: Blood Core, multiple Aragwaith units, Demon Grunt. * Assigned a more dignified generic portrait to Cron (1.11.x only). * Music and sound effects: * Mitigated [fade_out_music] causing a portion of the previous track to be heard at full volume at the end of the fade-out sequence. It still won't help in all cases. * Scenarios: * Fixed additional bugs with hero ellipses on Wesnoth 1.11.6 and later affecting Anya on every scenario and Durvan on scenario E3S7B and later. * Use STARTING_VILLAGES_ALL instead of STARTING_VILLAGES with large numbers to assign all villages to sides. * Skip inclusion of death events for characters that are not present during the first few scenarios of E1. * E1S3 - Civil War in the North: * Fixed the first defined on-map unit (usually Galas) becoming permanently invisible. * Minor prose tweaks. * E1S4 - Terror at Dusk: * Balancing changes to make the scenario easier on Wesnoth 1.11.7 and later, possibly connected to the new AI recruitment gold saving aspect introduced in Wesnoth 1.11.7 and enabled by default. * Minor prose tweaks. * E1S5 - Bay of Tirigaz: * Made it so Mal Keshar speaks for bat units when investigating shipwrecks. * Minor map tweaks. * Minor prose tweaks. * E1S6.1 - Quenoth Isle: * Fixed Elynia's ellipse reverting to a generic unit ellipse on 1.11.x during the faerie fire cutscene. * Not-so-minor prose tweaks. * E1S6.2 - Elves of a Different Land: * Not-so-minor prose tweaks. * Extended map for large screens. * E1S7 - The Search for the Past: * Improved ending cutscene transition. * Minor AI adjustments to make the undead minions recruit correctly on Wesnoth 1.11.7 and later. * Not-so-minor prose tweaks. * E1S7x - Resolutions: * Minor map tweaks. * Not-so-minor prose tweaks. * E1S8 - Fear: * Not-so-minor prose tweaks. * E1S9.1, E1S9.2, E1S9.3 - The Triad: * Not-so-minor prose tweaks. * Various cutscene improvements and changes. * E1S10 - Tears: * Minor prose tweaks. * E1S11.1 - Return to Wesmere, part 1: * Not-so-minor prose tweaks. * E1S11.2 - Return to Wesmere, part 2: * Fixed story text not appearing because of a missing macro inclusion (long-standing bug that's existed ever since the scenario was first released). * Minor map tweaks. * Minor prose tweaks. * E1S12 - The Queen: * Balancing changes. * Excluded time area for the E1S11.2 starting area on Wesnoth 1.11.7 and earlier (including 1.10.x) due to a bug with time area ids not being saved, resulting in a time area with local lighting that interfers with a cutscene sequence after reloading from a non-start-of-scenario save. * Fixed long-standing offset-by-one bug with a terrain mask applied near the end. * Minor map tweaks. * Not-so-minor prose tweaks. * Various cutscene improvements and changes. * E1S13 - Death and Rebirth: * Minor cutscene improvements and changes. * E2S1 - By the Moonlight: * Minor AI adjustments for Wesnoth 1.11.7 and later. * Not-so-minor prose tweaks. * Now the scenario lives up to its name. * E2S2 - The Heart Forest: * Fixed fog not being cleared correctly when Allyna first appears. * Made it so Allyna introduces herself once three of the five bandits have been killed rather than all of them. * Minor AI adjustments for Wesnoth 1.11.7 and later. * Minor prose tweaks. * E2S3.1 - Unrest in Raelthyn: * Minor prose tweaks. * E2S3.2 - Revelations: * Minor prose tweaks. * E2S4 - Shifting Allegiances: * Minor prose tweaks. * E2S5 - The Eastern Front: * Minor prose tweaks. * E2S6 - The Voyage Home: * Not-so-minor prose tweaks. * E2S7 - The Voyage Home: * Minor prose tweaks. * E2S8 - And then there was Chaos: * Minor AI adjustments for Wesnoth 1.11.7 and later. * Not-so-minor prose tweaks. * E2S9 - New Hive: * Minor prose tweaks. * E2S10 - The Betrayal: * Minor prose tweaks. * E2S11 - A Final Confrontation: * Maybe-minor prose tweaks. * Minor cutscene tweaks and improvements. * E2S12 - Fate: * Minor cutscene tweaks and improvements. * E3S0 - Opening (Within): * Minor cutscene tweaks and improvements. * E3S1 - Beyond her Smile (A Light in the Darkness): * Minor map tweaks. * Various cutscene improvements and changes. * E3S2.1 - Return to Raelthyn: * Minor map tweaks. * Increased initial gold supply for the second human player side. * E3S2.2 - Reckoning: * Minor prose tweaks. * E3S3 - Amidst the Ruins of Glamdrol: * Minor prose tweaks. * E3S4.1 - Outpost of Hell: * Minor prose tweaks. * E3S4.2 - Gateway: * Minor prose tweaks. * E3S5 - Pass of Sorrows: * Minor map tweaks. * Minor prose tweaks. * Minor ending cutscene improvements. * E3S6 - Divergence: * Minor prose tweaks. * E3S7A - Dark Fire: * Prevent crashing Wesnoth 1.11.8 due to a missing initial time of day (part 1 only). * Minor AI adjustments for Wesnoth 1.11.7 and later. * Minor map tweaks. * Minor prose tweaks. * E3S7B - Dark Sea: * Minor AI adjustments for Wesnoth 1.11.7 and later. * Minor prose tweaks. * E3S8A - Interim: * Prevent crashing Wesnoth 1.11.8 due to a missing initial time of day. * Minor cutscene improvements nobody could possibly notice. * Minor prose tweaks. * E3S8B - Destiny, part 1: * Minor aesthetic changes nobody could possibly notice. * Fixed parts of the map being unintentionally uncovered upon entering Hemérilyel's chamber. * Made Hemérilyel more aggressive towards the player on Wesnoth 1.11.2 and later. * E3S8C - Breakdown: * Minor AI adjustments for Wesnoth 1.11.7 and later. * Minor prose tweaks. * E3S8D - Destiny, part 2: * Minor prose tweaks. * E3S9 - Dark Depths: * Minor prose tweaks. * E3S10 - Blood: * Minor prose tweaks. * E3S11 - After the Storm: * Minor prose tweaks. * E3S12 - Destiny, part 3: * Minor prose tweaks. * E3S13 - Epilogue: * Minor prose tweaks. * Units: * Balancing changes: * Imps are now immune to the plague weapon special. * The Protection ability affects own units of any lower level again instead of only level 0 and 1. * Affected units: Demoness Hellbent Tide, Aragwaith Shield Guard, Aragwaith Ancient Banner. * Physical endurance no longer resets statuses (poisoned, slowed, etc.). * Decreased Lumeril Glyph Mistress' arcane damage resistance from -10% to -20%. * Decreased Fallen Faerie's cold ranged attack strength from 11-3 to 10-3. * New or improved unit animations: Verlissh Matrix Core, Shaxthal Custodian Drone, Shaxthal Queen, Verlissh Matrix Flow System, Verlissh Control Spire, multiple Aragwaith units, Dusk Faerie line. * Made it so the Falcon unit type and the lightfly movetype are only defined if the mainline Khalifate faction is not present, by testing the existence of core/units/khalifate/Falcon.cfg. * Fixed a minor inaccuracy at the beginning of the Terror ability description. * The spawn controller code (used e.g. in Shaxthal hives) has been completely rewritten in Lua. No behavior changes expected.
Merry Christmas/Happy Holidays, everyone!
After my successful extraction of the kate-wml-syntax portion of Wesnoth-UMC-Dev, the next goal obviously had to be just a little more complicated than that. Between Invasion from the Unknown and After the Storm, AtS has a relatively simple VCS and meta-history spanning only one main branch and one short-lived branch, as well as a number of tags based only on the main branch.
In reality, a tool called reposurgeon ought to be the most optimal for performing this task correctly and elegantly, but I have my reasons to avoid it. Nevertheless, if you need to convert your own SVN repositories to Git, it is probably what you should be checking out instead of these blog posts. I’m here to do things my own way and learn while at it; the
git svn portion of this procedure is very boring and nothing out of the ordinary, but
git filter-branch is a handy power tool that comes bundled with Git and which may prove useful to me for non-conversion tasks as well, and in fact, it already has done so. It’s worth noting that being a power tool, you do not want to put a clueless simpleton in charge of applying it on your repository; then again, if its documentation is anything to go by, reposurgeon is not much better in that regard. All this is probably for the best.
(I’m also taking this opportunity to express my absolute befuddlement at Git’s liberal
git push -f implementation, presumably a consequence of Git being originally designed for the Linux kernel’s mail-based workflow. While
git push -f can be useful under very specific circumstances, I have heard and seen people use it as a Git panacea without realizing the consequences.)
I already described the
git svn step in broad terms in Part II of this series, so I am not going to go into details here. The fun part continues to be the
git filter-branch (repository rewrite) operation, although this time we also need to deal with SVN tag branches.
As I had the perform the After the Storm repository rewrite multiple times on a remote server before I realized my desktop box’s superior disk I/O and tmpfs configuration and lack of console lag resulting from a slow SSH link would be a considerable advantage, I wrote a few scripts to help me throughout the whole procedure. I made several highly risky and specific concessions aimed solely at dealing with my particular use case, though, so I can’t possibly publish those scripts without either feeling ashamed of my subpar shell scripting skills or inadvertently allowing one of the aforementioned specimens to wreak havoc on somebody’s data, perhaps even their own! Either outcome would be regrettable.
Thus, I am providing a pseudocode outline of the rewrite procedure instead:
git svn’s point of view, HT is an empty commit object that incurs in no tree changes. This is an important thing to keep in mind for the next step.
Tagging add-on 'After the Storm', release <VERSION> from trunk, using r<REV>), I opted for an automatic minimalistic tag message for T of the form
After the Storm version <VERSION>, reminiscent of the
AtS: version <VERSION>commits preceding the tag branching point.
git-svn-idmetadata trail in the commit message so it reads e.g.
[Wesnoth-UMC-Dev SVN r12345]and strip extra empty lines between paragraphs.
/musicdirectory recursively if it appears in the commit’s tree.
git filter-branchdoes not do this by default for some reason, so you may end up with tags pointing to dangling objects otherwise.
git fsckto check that the repository isn’t broken at the end.
Part 1 was done through a “simple” sequence that extracts the needed information from every tag branch. At the risk of making myself eligible for the questionable classification from the start of this post, the code I used for this was:
tagproject="After the Storm" git for-each-ref --format='%(refname)' refs/heads/tags | cut -d / -f 4 | ( while read ref; do set -- `git log -n 1 --format="format:%ct %at" refs/heads/tags/$ref` export GIT_COMMITTER_DATE="$1 +0000" export GIT_AUTHOR_DATE="$2 +0000" set -- `git log -n 1 --format="format:%ae %an" refs/heads/tags/$ref` export GIT_AUTHOR_EMAIL="$1"; shift export GIT_AUTHOR_NAME="$*" set -- `git log -n 1 --format="format:%ce %cn" refs/heads/tags/$ref` export GIT_COMMITTER_EMAIL="$1"; shift export GIT_COMMITTER_NAME="$*" echo " * $ref (REF: `git rev-parse refs/heads/tags/$ref` CTS: $GIT_COMMITTER_DATE ATS: $GIT_AUTHOR_DATE CA: $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> CC: $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL>)" git tag -a "$ref" -m "$tagproject version $ref" "refs/heads/tags/$ref"; unset GIT_COMMITTER_DATE; unset GIT_COMMITTER_EMAIL; unset GIT_COMMITTER_NAME unset GIT_AUTHOR_DATE; unset GIT_AUTHOR_EMAIL; unset GIT_AUTHOR_NAME git branch -D "tags/$ref" > /dev/null done )
The most glaring issues with this code are that it invokes
git log multiple times for each commit (which could be solved if I were writing this pipeline for general usage, which I am not), and it does not take into account author/commit timezones; but it does not do so because
git svn does not either! Subversion normalizes dates to UTC, so
git svn cannot possibly know the original timezones used. Ideally, one would use
git filter-branch to rewrite these commits on part 2 (or even before, in fact) to convert their timestamps to local ones with a timezone offset attached, but determining the offsets to use involves knowing the authors/committers’ locations around the globe and any local daylight saving time regulations that may come into play depending on the local date and time. For a repository that has more than one committer or author, this is just pointless pedantry bordering on pathological perfectionism. I have a whole history with the latter and After the Storm which I would rather not repeat with something as mundane and short-lived as a one-time Subversion-to-Git conversion. Seriously.
Avoiding the kind of perfectionism that ruins lives is also the reason why I opted to leave commit messages intact save for the metadata trail instead of taking the opportunity to fix commits with long/inexistent summary lines, sloppy grammar/spelling/punctuation, commit messages consisting solely of a bullet list, and Kri’tan.
Compared to the tag preparation part, the actual
git filter-branch invocation is rather simple:
# Temporary dir on /tmp (a tmpfs mount) to trade disk access for RAM gittmp=`mktemp -uqd --tmpdir gittmp.XXXXXXXXXX` # Strip double blank lines and convert git-svn-id metadata trail # to a more readable format (e.g. [Wesnoth-UMC-Dev SVN r99999]). msgfilter="cat -s | sed -re 's/^git-svn-id.*@([0-9]+) .*\$/[Wesnoth-UMC-Dev SVN r\\1]/m'" # Remove external music/ dir indexfilter="git rm --cached --ignore-unmatch -q -r ./music/" # Needed to rewrite tags to point to the rewritten commits tagfilter="cat" git filter-branch -d $gittmp --prune-empty --msg-filter "$msgfilter" --index-filter "$indexfilter" --tag-name-filter "$tagfilter" -- --all"
To be honest, disk I/O on my desktop machine is good enough (with a 7200 rpm HDD on a SATA III link) that asking
git filter-branch to use a directory on a tmpfs mount doesn’t make a noticeable difference, but your mileage may definitely vary. AtS’ repository is not too large nor does it contain particularly a particularly large number of files. Moving the operation from a remote VPS to my desktop, on the other hand, did make the overall process about two times faster.
--tag-name-filter "cat" part was actually recommended by both
git filter-branch and its documentation, since otherwise tags are left untouched and stop making sense since they point to commits that have been rewritten and thus no longer have the original SHA1 hash or ancestry. The rest of the invocation is just pedestrian business involving
git rm and standard Unix tools; it merits no explanation whatsoever.
Well, the recursive
/music directory removal actually does require some justification. This Wesnoth add-on originally included a couple of large music track files (in Ogg Vorbis format) within it, thus they ended up as part of its SVN tree. Over time, however, this approach proved suboptimal both for me (uploading new versions of the add-on from a 3.5G mobile broadband link) and players, who had to download over 14 MiB or so every time instead of 7 MiB, the approximate base size of the add-on without music. Thus, at some point I deleted
/music from the SVN tree in favor of a separate rarely-updated add-on, but its history remained there. Since converting to Git involves starting from a clean slate in a way, I decided I might as well delete a portion of history that’s no longer relevant or used and may take up a significant portion of the packed repository data... or it may not; it’s not the point, either way.
The end result is now available on GitHub, and it’ll become official once AtS version 0.9.9 is released.
It is worth mentioning that past a certain point, power tools like
git filter-branch become an unfathomable corrupting force; if you spend too much time tweaking an invocation sequence and examining the results, this force seeps into your very soul and makes you do absolutely ridiculous things. It is no exaggeration when I say that such an outcome was but narrowly averted for my conversion of the AtS repository.
Anyway, the last SVN commit and the first native commit to the repository now look like this:
commit bbf8df680322ce585debb2b3068b446b87616371 Author: Ignacio R. Morelle <shadowm@xxxxxxxx> Date: Mon Nov 25 06:27:34 2013 -0300 Add a Markdown README.md containing general info, replaces BUGS commit 040ca5e81ef2a8851e331cfc5a4f59ae03d6bca2 Author: Ignacio R. Morelle <shadowm@xxxxxxxx> Date: Sat Nov 23 02:18:29 2013 +0000 AtS E1S4: mismatched tense fix [Wesnoth-UMC-Dev SVN r19373]
I opted for not stripping the
AtS: prefixes in past commit summaries (mostly those since 2011) used to identify Wesnoth-UMC-Dev projects. The reason is that I don’t really want to spend more time figuring out what words after the colon would then need to be excluded from capitalization aside from wesnoth-optipng. Laziness.
The whole of
/music was excised from the repository history, resulting in an overall size decrease from 25 MiB to 17 MiB. Given the number of PNG file additions and recompressions (“wesnoth-optipng pass” commits) taking place in the past, as well as the existence of some shorter Ogg Vorbis files under
/sounds, I do not think it’s feasible to shrink it any further without dropping additional relevant history. Again, the
/music removal was purely for practical purposes.
I guess you could say that balancing out my perfectionism with my unrelenting pragmatism and laziness was key to accomplishing this task within a finite amount of time.
Next target: Invasion from the Unknown. Since it contains oddities like changing tag branches and an unusual layout for its first commit (revision 1), it’s probably not going to be as easy as AtS.
Version 0.9.8 is out.
This is primarily a fast-tracked bugfix release which deals with the issue reported in the posts leading up to this in the Add-on Feedback topic.
The complete changelog for this version follows:
Version 0.9.8: -------------- * General: * Added a preload-time event to warn the player when attempting to use an unsupported Wesnoth version (< 1.9.10, < 1.10, = 1.11.0, = 1.11.1 at this time). * Scenarios: * E1S1 - The Skirmish: * Minor prose tweaks. * E1S3 - Civil War in the North: * Minor map tweaks. * Minor prose tweaks. * E3S4.1 - Outpost of Hell: * Fixed a side-effect of the [hidden_unit] clobbering fix from 0.9.7 that renders the scenario unwinnable after killing the final enemy boss. * Units: * Updated Chaos Gunner line animations for compatibility with Wesnoth 1.11.7. * Terrains: * Added litter terrain overlay from doofus-01, not currently used yet. * Code: <http://r.wesnoth.org/p555984> * Graphics: <http://r.wesnoth.org/p556000>
I posted a rather extensive report of the results of the
baldras.wesnoth.org migration to the developers’ ML because Ivanovic wanted me to post updates to the ML for some weird reason. I am fairly sure the people who are actually subscribed to it skip them entirely.
Because I care about transparency and stuff, I pointed out an important thing there:
[…] the migration process went smoothly and now the wiki is running MediaWiki 1.21.2 instead of 1.16.1. Look up 1.16.1's release date and the branch's EOL date and you'll understand why I've been saying "not really optimal" and "deplorable" when referring to the wiki.
(Only after sending the mail I realized that I wrote MW 1.16.1 instead of 1.15.1. The version that was running on
wesnoth.org is indeed 1.15.1, not 1.16.1.)
I don’t really have anything to add here, but I am providing a verbatim copy of the email after the break anyway, only because it’s a wall of text and I like those.
As announced in the Wesnoth.org forums, today (although it’s still yesterday for me) we are moving most of Wesnoth.org’s facilities to a new host, to continue the migration process that started with the previous move of
What you probably wouldn’t guess from the announcement is how complicated the underlying machinery actually is. I did not want to clog up the post with additional details because that is often confusing for most people, but here is a list of services that will be directly affected by this whole operation:
www.wesnoth.org— Static front page, downtime notices, wiki images
wiki.wesnoth.org— MediaWiki application, site stylesheets
forums.wesnoth.org— phpBB application, forum attachments
r.wesnoth.org— Short forum URLs
servers.wesnoth.org— wesnothd application instances for Wesnoth 1.10 (stable), 1.11 (development), 1.8 (oldstable), 1.6 (ancientstable), and Git master/version redirection
wesnothd.wesnoth.org— wesnothd statistics for the current stable, development, and oldstable server instances
gettext.wesnoth.org— Translation statistics for mainline and user-made (WesCamp-i18n) content
eclipse.wesnoth.org— Timotei’s IDE thing and stuff
units.wesnoth.org— Automatically-generated mainline and UMC unit trees
replays.wesnoth.org— Archive of replays of MP games with observers enabled
And the redirection rules for various subdomain aliases:
wiki.w.o]— Multiplayer Code of Conduct
forums.w.o] — TGT tournament topic in Multiplayer Development
wiki.w.o] — Game manual
wiki.w.o] — Developer-oriented resources
github.com] — GitHub repository
github.com] — Release changelogs
gna.org] — Issue tracker for the Gna.org project
gna.org] — SVN repository at Gna.org
Because we are seriously understaffed, I have literally spent months copying things over, testing, upgrading, rebuilding, and even redoing some steps several times in some cases. However, only during the last month I have been working non-stop on stuff, realizing the deplorable operating conditions of a couple of our services, one of which I cannot fix on my own.
Today’s migration doubles as a full OS upgrade accompanied by an extensive internal network architecture change, as well as an upgrade of the forums and wiki applications. Upgrading the forums has never been a big deal for me, but everything else is uncharted territory. Given these complications, I have spent most of the aforementioned amount of time learning and testing things that shouldn’t really be my business in an ideal world, MediaWiki included. It’s not a particularly rewarding task (aside from the learning experience), but it must be done for the sake of all... especially that of our finances department.
I’m awfully anxious about all this. I’m going to make sure to have a cup of tea handy when the time comes.
Because the Wesnoth.org forums people keep addressing me as ‘shadowmaster’ or variations thereof:
And that’s it. The ‘m’ no longer means anything, so the username has essentially come full circle.
(All names in this post are case-sensitive.)