Autour des méthodologies de développement logiciel

Un article posté sur Hacker News m’a entraîné dans une suite de développement sur les méthodologies de développement logiciel.

La problématique est la suivante : comment peut-on industrialiser le développement logiciel – c’est à dire faire en sorte d’obtenir des résultats prédictibles en terme de temps et de qualité ?

Et son corollaire : parmi les multiples méthodes de développement logiciel qui n’ont pas manqué d’être proposées depuis quarante ans, pourquoi est-ce qu’aucune ne semble atteindre ce but ?

Le point de départ est un article de Ian Miell, My 20-Year Experience of Software Development Methodologies.

Sa thèse est que toutes les méthodes de développement qu’il a rencontré dans sa vie professionnelle (Waterfall, Flow, Agile, ou n’importe quoi de vaguement indéfini) sont des fictions collectives. Ces méthodes ne sont pas bonnes ou mauvaises en soi – mais permettent de s’organiser en grands groupes suivant différents principes partagés par tous.

Lean, Agile, Waterfall, whatever, the fact is we need some kind of common ideology to co-operate in large numbers. None of them are evil, so it’s not like you’re picking racism over socialism or something. Whichever one you pick is not going to reflect the reality, but if you expect perfection you will be disappointed.

Au passage, un lien pointe vers cet autre article d’il y a quelques années, Why don’t software development methodologies work?. En résumé, l’article dit ceci :

Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion. The resulting passive-aggression kills productivity faster than any methodology or technology decision.

Si pour lui par essence aucune méthodologie de développement logiciel ne fonctionne, par sédimentation des rituels, il voit quand même un critère qui permet de prédire le degré de succès d’une équipe de développement :

My own experience, validated by Cockburn’s thesis and Frederick Brooks in No Silver Bullet, is that software development projects succeed when the key people on the team share a common vision, what Brooks calls “conceptual integrity.” This doesn’t arise from any particular methodology, and can happen in the absence of anything resembling a process. I know the feeling working on a team where everyone clicks and things just get done.

Ce qui me mène vers des recherches effectuées par des employés de Google, sur d’autres critères qui influencent la performance d’une équipe de développement.

Leur prémisse est que le succès d’un projet dépend plus de l’équipe que de la méthodologie employée. Et d’après l’expérience des auteurs, cinq critères semblent être prédominants :

  1. Psychological safety: Can we take risks on this team without feeling insecure or embarrassed?
  2. Dependability: Can we count on each other to do high quality work on time?
  3. Structure & clarity: Are goals, roles, and execution plans on our team clear?
  4. Meaning of work: Are we working on something that is personally important for each of us?
  5. Impact of work: Do we fundamentally believe that the work we’re doing matters?

Je ne peux parler que pour moi, mais j’ai effectivement retrouvé tous ces critères dans les boulots où je me suis vraiment plu. Et inversement, dans les endroits ou les moments plus disfonctionnels, je retrouve bien ce qui manquait dans les éléments de cette liste. Comme quoi…

Link’s Awakening disassembly progress report – week 3

This article is part of an ongoing “Disassembling Link’s Awakening” series, where I attempt to gain some understanding on how special effects where implemented in this game.

This week didn’t get as much work done as the previous week—but it sure has some progress.

Export all labels to the debug symbols.

rgbasm has special rules for deciding whether or not a label should be exported to the debug symbols. By default, only global labels (that are visibles by all code units) are exported into the .sym file.

This makes us face an annoying choice when labeling code. The best way to label intermediary routines would be to use scoped labels, which are prefixed with a dot, like .some_label. Scoped labels hey don’t leak into the global namespace, and there can be several of them with the same name (as long as they are not in the same scope). But in this case, these labels won’t get exported to the debug symbols, and won’t show up in our disassembling tools.

To work around this, we had to resort to an annoying fix: make most of the labels globals, like SomeLabel::. But then we loose the locality of the label, and all of them have to be unique in the whole source of the game.

Fortunately, the contributors who maintain the rgbds toolchain recently added an option to export all labels to the debug symbols, regardless of the label visibility. We can now have our cake and eat it: use local labels, and still having them visible in the debugger. Liberal use of local labels makes the disassembly much more readable, and I’m quite happy to have this technical restriction lifted.

BGB screenshot showing all labels in the disassembly code

Having all labels displayed in the debugger makes the disassembly much more readable.

Label the File Save dialog routines.

The code for the File Save dialog is located right at the start of bank1. As this is code I often stumble upon when opening this file, I was keep to label at least the general structure of it.

As many other places in the code, it uses a jump table to control the transition between the Overworld and the File Save dialog.

Link's Awakening File Save dialog

Label more of the render loop

For one year I’ve been writing the draft for an article that details the structure of the game’s main render loop. Unfortunately some bits and pieces of this section of the code are still obscure to me.

This week I tried to fill the gaps in areas I don’t understand yet. And a lot of progress has been made, especially on the transition special effets.

During the rendering, several types of special effects can affect the background, and give it a wavy feeling. Initially I though there were only three effects:

But it turns out other transitions are using this code! They actually count in reverse around zero:

Unlike the first three transitions, these one are interactive: instead of the same frozen frame being displayed while the transition effect is playing, during interactive transitions the game continues rendering frames—which makes it possible to move Link while the special effect is applied, for instance.

The Wind Fish apparition

Without this interactive effect, the Wind Fish apparition wouldn’t be as impressive.

What’s next

At some point I’d like to finally complete this article about the main render loop structure. There are still a few lines of mystery, but it’s almost there!

Link’s Awakening disassembly progress report – week 2

This article is part of an ongoing “Disassembling Link’s Awakening” series, where I attempt to gain some understanding on how special effects where implemented in this game.

This week got quite some work done. The entire introduction sequence is now labeled, and the Awake disassembler starts to get quite useful.

This progress report has less graphics and screenshots than last week, as most of this week changes are done to the code. But let’s have a look anyway.

Export High-RAM variables to debug symbols

Last week, we saw how refactoring the disassembled code allowed the name of variables stored in WRAM to be exported to the debug symbols when compiling the game. This was useful for most game-related variables, which are stored in WRAM.

But there is another range of memory that was avoided until now: the High-RAM.

On the Game Boy, the High-RAM (or HRAM) spans from $FF80 to $FFFE. It is sometimes named “the zero page”, as it is the first range of addressable values.

What makes HRAM different? Well, accesses to HRAM are slightly faster than for the rest of the RAM. This is because of a shorter variant of the ld instruction: if only a single byte of addressing is provided, the load or store will be performed from the HRAM. As it saves a byte by not specifying the entire address, this variant is slightly faster. Also, the HRAM is the only memory that can be addressed during some specific operations—like a DMA transfer.

A mapping of the memory on the Game Boy hardware The HRAM (or “zero page”) is located at the end of the magenta section.

So great, we create a new section for the HRAM area, map the variables names (as we have done for WRAM), and we’re done, right?

Not so fast.

Let’s recap: if we want HRAM constants to be exported into the debug symbols, we need to turn them from constants into labels. For this, we need to define a custom section in the assembly code, to tell the linker where our symbols have to be exported in the final binary. And if we use a custom section, we hit a known limitation of the rgbds toolchain:

If you use this method of allocating HRAM, the assembler will NOT choose the short addressing mode in the LD instructions LD [$FF00+n8],A and LD A,[$FF00+n8] because the actual address calculation is done by the linker.

If you find this undesirable you can use the LDH [$FF00+n8],A and LDH A,[$FF00+n8] syntax instead. This forces the assembler to emit the correct instruction and the linker to check if the value is in the correct range.

(From the RGBDS documentation)

This means we have to convert every load instruction referencing HRAM variables from ld instructions to ldh instructions.

Fortunately this is easily done with a simple regular expression, which is what was done. And so with a bit more work, we finally get HRAM variables exported as well, and displayed in our disassembling tools.

Label the initialization sequence

When booting up the Game Boy, the internal firmware executes some hardware initialization—then hands over control to the game itself. And the first thing the game does is initializing itself.

Although some of this process was documented, this week the whole initialization sequence has been entirely labeled.

This initialization sequence is rather short, and quite readable. It involves:

An interesting tidbit is hidden in the routine that initializes the save files. When compiling the ROM with debug tools enabled, it will write a pre-defined save file to the first slot. This save file has every item and quest status unlocked, all hearts, all seashells, and so on. Quite useful for testing features that appear later in the game.

Investing in tools

When disassembling a binary, a good tooling is key. A good disassembly browser can make reading, following and understanding the code much easier. A good memory viewer will help to tell quickly whether a particular piece of data contains graphics, a map, or some music. A good debugger that can set fine-grained breakpoints will help understanding when the code attempts to access some specific memory location. And professional disassemblers are large and expensive software, because they can get so much done.

Since last week, some tooling improvements were merged. Notably, Awake can now import debug symbols generated when compiling the game.

Demo of importing debug symbols in Awake Importing debug symbols turns meaningless addresses into beautiful procedure names.

This paid off quickly, and made understanding the initialization sequence code much easier.

The following issues in Awake were also fixed:

I feel more confident with awake’s code base now, but there are still many parts that I don’t understand well. The hairiest piece of code seems to be the procedure flow analysis, which turns assembly code into C-style statements—and I’d like to understand it better, and start to improve it in some ways.

What’s next

For the next weeks I’d like to focus on the main render loop. It is already partially labeled, but many parts are still obscure to me. Hopefully the new tooling will make it easier to understand what is going on!

Link’s Awakening disassembly progress report – week 1

This article is part of an ongoing “Disassembling Link’s Awakening” series, where I attempt to gain some understanding on how special effects where implemented in this game.

This week I found some time to work again of this Zelda: Link’s Awakening disassembly project. And it got quite a few improvements done!

Export all the graphics

The various graphics used in the game are stored in the ROM as binary files. Until recently, the binary sections containing the pictures weren’t even entirely identified.

This week, the remaining graphic banks were all mapped out, and all of them are now exported as PNG files. When compiling the game, the PNG files are converted back to their original 2bpp format, and inserted into the final binary.

For now, most of the pictures are exported as large sheets containing multiple graphical elements. For instance, this is a portion of the graphics sheet for the dungeons graphics.

A portion of Link’s Awakening dungeons graphics

Of course, it is possible to split these elements into individual items. But for now only a minority some background tiles and sprites are split up individually.

Link’s Awakening title Link’s Awakening palm trees Link’s Awakening sea foam Link’s Awakening Windfish egg

There is still work to do there.

You’ll also notice that most of the graphics are actually stored twice. Now remember that Zelda DX is actually a port on the Game Boy Color of the original “Zelda:Link’s Awakening”, which was initially released on the original monochrome Game Boy. However the remake could still run on the original Game Boy.

So it seems that during the development of the Game Boy Color remake, Nintendo’s developers duplicated all the graphics. This allowed them to tweak the graphics for the color edition—while leaving the original grayscale elements intact when running on a Game Boy or Game Boy Pocket.

You can see below some slight differences of graphics between the original graphics and colored ones.

Link’s Awakening overworld tiles

N.B. All graphics, colored or not, are stored as grayscale images. Color is applied at runtime, using external palette data. This is what allows an identical piece of graphics to get various colors, depending on the context it is used.

Although the gameplay and story of the remake were the same than in the original game, some elements exclusive to the Game Boy Color were added. A notable one is the appearance of a photographer, which will sometime appear during the game to take a picture of Link’s adventures.

Good news: these pictures have been extracted too! However they are often compressed in some way. Space on a cartridge isn’t cheap, and was to be used with precaution. Also the photography scenes sometime contain moving parts and dynamic elements—which had to be encoded into the picture somehow.

This is why, although some of these pictures are recognizable, many of them look like garbled content, or a mixture of different items.

First photo you obtain in the game

Memory locations now show in the debugger

Disassembling the game is mostly about making sense of a bunch of machine code. And one of the most used technique is to reconstruct the labels and comments of the source code.

One good thing about labeling the different parts of the disassembled code is that they’ll get exported into debug symbols. These debug symbols can be read by emulators, debuggers, and other different tools to give meaning to the game code. The more we label, the more we can use these labels to progress further.

But until now, only function labels were exported into debug symbols. But how about the memory location labels, which indicate how are used some specific part of the memory, and are almost as important as function labels? Well, due to the way they were labeled in the disassembly, memory labels were not exported at all.

This week the declaration of the memory labels was rewritten, so that they get exported into the debug symbols. Which means for instance that memory locations now show up in the debugger! This will definitely make reverse-engineering easier in the future.

BGB debugger without the memory location labels BGB debugger with the memory location labels

Awake decompiler

A few years ago, Github user @devdri started working on a tool to help the Game Boy games disassembling efforts : a custom Game Boy decompiler named awake.

The idea is to present the game code into a nice user interface, which allows browsing the code and jumping back and forth easily. It also features many heuristics to make sense of the assembly code: detecting the start and end of procedures, exposing jump tables, and displaying the assembly code as high-level C-style expressions.

Although this project hasn’t been active for the past few years, it seems it could be quite useful on its own. A few missing features would be a nice improvement to this tool though.

So this week I added support for labeling functions and memory locations from the UI. Support for importing debug symbols is also coming soon. Hopefully this will make awake a worthwhile tool for understanding the disassembled code.

Screenshot of the awake Game Boy disassembler

Disassembling guides

It’s not easy to jump into a new project. How is the game even compiled? Do I need a custom toolchain? What are the tools used to explore the code and graphics?

Fortunately some guides are now included in the project README file. You’ll get a quick overview on how to use a debugger to label code, and how the scripts used to extract the graphics data work.

What’s next

For the next weeks I would like to:

And we’ll see how it goes.

🚀 Partager les succès

Si vous travaillez sur un projet dans une petite équipe, il est sans doute facile de suivre ce qui se passe en ce moment. Qui fait quoi, quels sont les sujets du moment, quels sont les dernières modifications visibles et les dernières étapes effectués.

Mais au fur et à mesure que l’équipe grandit, ce sens de « qu’est-ce qui se passe en ce moment » devient plus difficile à appréhender. Et on découvre un jour qu’on n’était pas au courant de telle nouveauté, de tel changement dans l’équipe d’à côté, ou de tel recrutement. Ça a été annoncé, sûrement – mais peut-être pas aux bonnes personnes, ou au bon moment.

Chez Capitaine Train, on avait mis au point une petite convention simple pour rendre les succès plus visible.

“There are things that can give you wings, but I’ll give you a rocket!”

Pour partager les succès, pas besoin d’en faire des caisses. Ce qu’on faisait, c’est simplement de poster un message dans un canal public sur le chat interne (Slack, en l’occurrence).

Une seule règle : mettre un emoji :rocket: 🚀 au début du message.

C’est tout.

Pas besoin d’être bien long ; il s’agit juste de communiquer aux autres une nouveauté dont vous êtes heureux ou fier :

Comme c’est très léger, ça ne prend pas beaucoup de temps à écrire.
Comme c’est rapide, on hésite pas à annoncer la petite nouveauté dont on est fier, même si ce n’est qu’un petit détail.
Comme c’est juste une convention, ça ne rajoute pas un processus de plus.
Et comme c’est sur Slack, ça permet de recueillir les félicitations et les applaudissements (et c’est important, les applaudissements).

Composer les outils pour aller plus loin

La beauté des conventions, c’est que ça active plein de possibilités. On peut s’en servir pour faire plein de belles choses en plus.

Par exemple, pour les changements qui s’y prêtent, on peut ajouter une image à son annonce : une capture d’écran, la photo de la personne recrutée, ou un gif animé qui présente une nouvelle fonctionnalité.

Pour être notifié de ces annonces spécifiquement, il est possible de rajouter :rocket: à la liste des mots-clefs qui déclenchent une notification. Ça peut permettre de mieux filtrer le bruit, et de n’être notifié que de ce qui est important.

Enfin, il est assez simple de générer une newsletter interne, mensuelle ou bimensuelle, à partir de ces annonces. Une simple recherche sur 🚀 renverra tous les messages à inclure dans la newsletter. Il est même possible d’automatiser le processus : coder un petit bot pour Slack qui recopie les :rockets: dans une planche Trello ; et un autre bot qui, une fois les annonces mises au propre, envoie automatiquement une newsletter avec le contenu de la planche.