Retrogaming woes (with patches for some)

Hi all,

Not sure if this is where I should send this, my gut instinct would’ve been to just shoot an e-mail to the developer, but the FAQ says to “not report issues by email” and the #lutris IRC channel cannot be joined, so I’ll try here. I mostly use Lutris to run games from old platforms, and in doing so I’ve noticed a few issues, so I made some patches for them (they’re made against version 0.5.5), which you can get from my cloud, I’ll briefly describe them below.

First, here’s an annoyance: Since the 0.5.5 release, I couldn’t use the Install buttons on the website to install games, I was told that the lutris binary was not found. Upon noticing the binary was in an unusual location (/usr/games. It’s in my $PATH but I guess Chromium or xdg-open or whoever handle those links just hasn’t got it in theirs), I remembered the previous Debian package had more files in it. So, the solution was a quick one. Get the 0.5.5 source, create a file named debian/lutris.links with the following content:

/usr/games/lutris  /usr/bin/lutris

And rebuild the package with the usual debuild -uc -us invocation. There you go (for those who are wondering: The culprit is commit af070229db4936e4c9cd334b275a545b07882c45 from December 26)!

For everything else below, you can apply the patches to the Debian package quite easily by dropping the attached archive’s patches directory into the Lutris source’s debian directory, then doing a echo "3.0 (quilt)" > debian/source/format, and rebuilding the package the normal way.

So, to begin with the litany of little issues, we have the runner’s platforms: They’re, huh, a bit wonky. For instance:

  • The mednafen runner returns platform names which do not match the corresponding icon names and mixes games for different machines under the same platform, e.g., Nintendo Game Boy (Color) should actually be two platforms to have the correct icons show
  • The openmsx runner also declares a platforn name which does not match the icon (maybe not a problem, I didn’t see this runner used on the site)
  • The mess runner does not declare the SNK Neo Geo Pocket Color, even though it’s used for the NGPC installation scripts
  • The osmose runner does not declare the Sega Game Gear platform
  • The worst might be the libretro runner: It declares having cores for a plethora of platforms, but some of them are just missing from the runner repository (hxxp://lutris.net/files/runners/), for instance the opera core (for 3DO) or the kronos core (for Sega Saturn). And yes, looking at games on the site, those cores are used by installation scripts, which tells me they won’t work upon install as the correponding core won’t be downloaded (also, some cores which are available on the repository are not declared, like DOSBox).

The first patch in the archive (patches/0010-fix_runner_platforms.patch) fixes this, and standardizes all the platform names across runners. It also tries to return correct platforms for multiplatform runners/cores by applying a crude heuristic on the ROM filename (lousy, I know, but that was the easy option). For libretro, it also maps removed cores to ones that are still present (well, when there’s one, at least. To the guy who uploaded those J2ME mobile games: Sorry, you’re out of luck). Finally, it changes some platforms I think don’t really apply to the runners (I mean, come on, sure: ScummVM games might have been rereleased for Linux lately, but I’m sure most people would identify Monkey Island as an MS-DOS game!). You might also want to apply patches/0050-show_old_platforms.patch (which merges the platforms from the runners with those from the SQLite DB) to ensure platforms don’t disappear from the sidebar after applying the previous patch.

Next, I noticed an issue with the default installation directory format: It’s just the game title, which is all well and good, until you have the same game for different platforms, as it will mix the game files, and (what I was most anxious about) probably delete all the files for both versions if I uninstalled one. The file patches/0020-use_slug_for_install_dir.patch fixes this by using the full versioned slug. Also, the file patches/0030-add_directory_field.patch provides a directory field which you can use to change the directory a game is installed in (you’ll still have to move the files yourself, though). Another annoyance was that, upon installing a game for a new platform, the platform wouldn’t show in the sidebar until restarting the app. The file patches/0060-update_platforms_on_install.patch takes care of that.

Let’s go on with some more issues installing random games. Their scripts are, well, just broken. Take the Super Nintendo version of Final Fantasy III, for instance. The script says:

install:
- dst: $GAMEDIR
  merge: null
  src: rom

Guess where the problem is? That will NOT install the game to $GAMEDIR, instead the ROM will be left where it is, which can lead to some amount of head-scratching when that location is wiped (yes, I’d put the file under /tmp for the install). The script should read:

installer:
- merge:
    src: rom
    dst: $GAMEDIR

So, I made a patch (patches/0040-handle_broken_scripts.patch), but I guess the correct solution would be for someone to fix the script on the website.

Another example: Looking at the available Z-Machine games, I tried to install Arthur Yahtzee: The Curse of Hell’s Cheesecake. The game installed, but then failed to run, the frotz runner complained it couldn’t find the file, even though I could see it. So, what did happen? Well, the script just provides a naked filename, while the runner expects a full path. Applying patches/0070-frotz_default_dir.patch fixed the issue for me.

I also had much more trouble than expected with some runners:

  • Mednafen would run fine, but then freeze the KDE 5 Plasma shell upon exit, didn’t find any useful info about this problem, maybe some issue with powerdevil, I don’t know :+(
  • DGen would go fullscreen even though fullscreen wasn’t enabled. I even managed to completely mess my screen resolution trying to troubleshoot it
  • Osmose just would not run, instead failing with libQtOpenGL.so.4: cannot open shared object file: No such file or directory (is that a KDE4 library?! Definitely out-of-date!)

In the end, I just had enough of this (not to mention, each time you use another runner, you’ve to redo your joystick setup, and so on) and decided that, since Retroarch was working quite well, I would simply use it for everything I could. If you too have trouble with some emulators and don’t want to tediously change every install script, you might find patches/0090-replace_runners_with_libretro.patch useful: Assuming the runner platform patch above has been applied, it will run games for most emulators with Retroarch instead. Much better that way (at least for me)!

[edit: looking at the website now, it seems that most of the games have been switched to libretro, I can’t find any game run by either dgen or osmose, and pretty much none on Mednafen with the notable exception of Super Mario Bros., while I still have a lot of them, according to a SELECT COUNT(*) FROM games WHERE runner IN ('dgen','mednafen','osmose'). I guess someone had enough just like me and changed the scripts, so I might just be able to drop the above patch if I can find how to fix the games config files]

There are a few more patches, but they aren’t really fixes per se, rather some experimental enhancements. The first (patches/0080-frotz_alt_engine.patch) provides an option to enable a different engine for Z-Machine games, based on a patched version of jsZip. This is defintely hackish: In addition to applying the patch, it also requires dropping the runners/jszip-master directory into ~/.local/share/lutris/runners. And it needs chromium-browser to be installed (or to edit jszip.sh to change that). But if you like playing IF comfortably, you might like the fact that you get an automap feature (yay!) and autocomplete to boot (try <Tab> at the interpreter’s prompt).

Also, I found the installation process for most emulated games to be tedious: You have to navigate to your ROM directory for the console, which can be nested quite deep, and select the ROM file (which might have a completely different name, see below for my thoughts on the site’s game database), hoping you selected the right dump. So in patches/0100-rom_auto_selector.patch, I try another approach: Just select a directory, and it will recurse into that directory, making CRC32s of each file and comparing them to a CRC in the script. Once the file is found, it’ll be automatically selected. For this to work, the installation script must have something like this:

files:
  - rom: S/C:61F9E7CD:Select the game's ROM file

Note the CRC value before the message to display.

Well, that’s enough monkeypatching for today, I hope someone finds one of those useful. That being said, a problem I cannot patch so easily is the lackluster quality of the Lutris games database. There are quite a few issues (wrong naming, duplicates, etc.), but the one that makes me go bonkers is the fact that the platforms a game is advertised for do not match the platform you can actually install the game for. You don’t have to search very far to encounter instances of this, for instance the MSX platform has a total of 49 games (which is far from being a complete listing, by the way). But looking at each game in turn, only three (3D Tennis, Penguin Adventure and Le Mans 2) have an install script for MSX. So that was three clicks out of forty-nine which yielded something useful. I’m aware this paragraph sounds really negative, and I’m sorry for that, but honestly, this kind of kinks does not provide for a good user experience; filtering out the platforms that cannot be installed would be a Really Good Thing™, and I’d really appreciate if someone could fix that (I would do it myself if I could but as I said, the issue is server-side)…

Also: On the Runners (hxxps://lutris.net/runners) page, there are links to see what games a specific runner (or a specific runner version) has got. That’s nice… except it’s broken: Following any version number link will just show a blank page. Following the “Browse games for all versions” will only show the first 25 games with no pagination. Again, not very useful :-/

So yeah, I definitely like Lutris, the idea of converging all your gaming-related things in a single place is appealing, the UI is nice, and some parts actually remove hassle from the user (I’m mainly thinking about WINE here), but there are other parts that bother me somewhat (like runner binaries not being integrated in my package manager) and some which are hugely annoying (like the database issues above). That being said, it’s still a huge improvement over the previous situation re: gaming on Linux. So thanks you all anyway!

It says so because there’s a proper bugtracker provided by github.

If you haven’t rebooted since update it might be an issue with the system not refreshing paths correctly. Or maybe xdg-open doesn’t like /usr/games/bin.

Has Plasma ever been actually stable?…

That’s a Qt4 library (the Qt5 one is called libQt5OpenGL). KDE is just a DE.
Either way, it’s strange; the source hasn’t been updated for years but the readme clearly says Qt5 is supported (incidently it appears to be maintained by Lutris team). This one definitely should be reported as a bug (in the osmose bugtracker, probably… unless that’s an issue specifically with the build used by Lutris); though you may want to see if symlinking the Qt5 library to the requested name works.

Does this prevent the user from trying ROMs with different CRCs? That would be a rather backwards “feature”…
Also, why not SHA?

If you’re talking about platform tags, they’re set for platforms the game has been released for. And, since installers are entirely community-made, making install scripts for them is basically your (the user’s) part to do. Or if you’re too lazy for that, just create the game link manually, you don’t need an installer script to tell an emulator where the game ROM is.

Well as I said, these games can be installed for those platforms, the lack of userscript is because no one cared enough to add it (if you need it, you can add it). And from my perspective, searching for MSX games and only finding three would be much more frustrating than finding fifty with some of them not having a one-click install script (especially if my purpose of searching was to add missing install scripts).

AFAIK, that bugtracker requires you to sign up for GitHub (at least, it asked me when I hit the “new issue" button). Signing up for GitHub requires the user to accept terms such as:

We store and process the information that we collect in the United States in accordance with this Privacy Statement

(From GitHub’s Privacy Statement. I respectfully decline to have my personal information transferred to the US, especially post-Snowden)

Or that little nugget, tucked at the end of their ToS:

You and GitHub agree to submit to the exclusive jurisdiction and venue of the courts located in the City and County of San Francisco, California

I definitely refuse to allow some US corporation to sue me in remote locales an ocean away just to report a bug. That being said, if you know of a way to use the GitHub bugtracker without opening an account there and without sending any personal data over to the US, please tell, I’ll be glad to have a better way to report issues than a forum (I agree this is not the best venue for that).

Ha! I won’t fall for that trollish bait :+)

Well, I guess I could have several CRCs in that field, split them into a Python list and compare to that, I’ll have a go at it. Mainly, the goal is to make the installer more foolproof (e.g., avoiding ROMs which are bad dumps).

More computationally-intensive, if the user chooses a large directory I fear that could make the search quite slow, and since the file chooser widget I’m abusing for the purpose has no progress bar…

I’d figured that, but while such an information is interesting to have (even though, honestly, I’d just head to MobyGames or similar for authoritative info about a game), what I disagree with is their use for the search filters: When filtering for a platform, I expect to see the games Lutris will let me install quickly on that platform…

I’m definitely not complaining about the absence of scripts, I’d just like to have a way to filter for those who have already been written (there’s a “version” label next to each script, maybe there is some way to filter for those labels?).

Sure, the current scripts don’t do much more than tell what runner to use and ask you for the file. That’s why I wrote my proof-of-concept file searcher: I think it would be great if Lutris could help a bit more here.

Technically the only information you’re providing is the email which you use for login, your current external IP, and the contents of your bug report (unless you’re feeling like filling up user info any further). Which, considering where the servers are likely located, would be transferred to USA either way.
And, if you’re feeling particularly paranoid, nobody is stopping you from using Tor to access the site and a new email for registering :slight_smile:

I’m actually genuinely curious. I’ve used KDE when it was at v3.5; and stopped using it soon after trying v4. The meme about “Plasma not crashing” appeared way back then, and from my occasional attempts to check if they fixed it since, it doesn’t seem like there’s been much progress with that :smiley:

…You seem to be missing the point here: a Linux user should be able to use whichever ROM he pleases when he wants to do so whether it’s in a list approved by you or not.

I suspect you can display the progress bar outside of the file chooser (you’re not running the search when the dialog is still open, are you?).

I really doubt it’s an appropriate default in what’s essentially a community-maintained database (as in, Web 2.0). If at some point Lutris becomes insanely popular, and all related install scripts will be created within minutes of adding the game to database, then it’d make sense to create a separate filter for those willing to write the installers for the games which don’t have them; but as of now, the only feasible way to handle this is “you want it, you add it yourself”.

The “version” thing is a freeform string, so it’s hardly possible to make a reliable filter based on it. On the other hand, it might be possible to create a filter by runners in existing install scripts (though they’ll likely have to migrate the whole database to generate the data for it)… If you want the devs to implement that, you may have to make a feature request for it on their bugtracker :upside_down_face:

Well, I already use Tor regularly, I guess I can use it for that too, then. I still think that an open source project forcing users to signup for a closed-source service like GitHub is not necessarily the most open option.

KDE 4 was indeed a crapfest (at least the first iterations), it was absolutely unbearable. Since then, things have gotten way better, and I’m quite happy with the stability on the current version, I hardly had a crash or hard lockup since a long time (the “freeze” I was alluding to in my first message was rather a big slowdown, I still could access the ALT+F2 prompt to restart plasmashell). There are still things which do bother me (Akonadi being the most notable showstopper. That one sure loves crashing; thankfully, I only use KMail at work so at least it only causes me trouble half the time lol) but overall I’m pretty satisfied. That being said, I assume others might have a different experience: For instance, I specifically picked a machine with AMD graphics to have a supported open-source driver, maybe the compositor will not work as well for people with nVIDIA GPUs…

Two reasons why it would help to check the file:

  • If I choose a ROM which is for the wrong game, or even worse the wrong platform (so it just will fail to run later), it would be helpful if the installer told me I made a mistake
  • If my dump is bad, it would be much better for the installer to tell me right from the start, rather than let me find out later when the game crashes, don’t you think?

Which is, by the way, the reason why databases such as TOSEC provide CRC and MD5/SHA1 checksums. I’m mentioning that database specifically, because the website has API endpoints to query an imported TOSEC collection.

Right, I actually based my code on InstallerWindow::ask_user_for_file() which didn’t create a progress bar, I’ll add it (or maybe not, since I gather from your previous answer that the installer shouldn’t have that kind of checking/search feature, so no need to code this any further if no one wants it).

I still think a way to search for games having an installer with a specific “version” field would be useful, and not too difficult to implement (mostly getting a list of versions is just a matter of running a SELECT DISTINCT on that field. Although I never used Django to build a website, I’m more of a Laravel guy, so I might be wrong here)…

Like I said before, that one is already present on the runners page, for instance here are the Libretro games. But it misses the pagination…

Don’t they read this forum (since you answered, I thought you were one of them actually)? And isn’t there really any other way to contact them by open decentralized protocols (i.e., a mailing list, Jabber etc.) rather than centralized services like GitHub?

Well, yeah; in my experience, the only thing it’s been stable at is crashing within the first few hours of normal usage :smiley:
Though due to some circumstances, I wasn’t able to get a machine with a modern AMD videocard until recently (and yes, it indeed feels great when a Wine game doesn’t hang up after an Alt+Tab); still, I had problems even when using Intel.

Again, the point is not what would be helpful, it’s what the user is allowed to do. You seem to believe that the only case where a CRC would differ from yours is when the file is messed up, but I’m pretty sure that, for instance, localized versions of games (of which there would likely be a lot, at least locally distributed) would also mismatch. And I guess if someone felt like hacking the game for whatever reason (like cheating during a really hard boss fight, or just because of generally sucking at playing), it would really annoy him if some software didn’t let him use it because “it’s a bad dump”.

What it shouldn’t do is enforcing it; if it’s an option (like a panel right under the regular file selector) which the user can simply ignore, it probably won’t be much of a problem.

If you’re talking about a proper version, there’s no point in doing a search by that (as it’s something individual to the script itself and is of no consequence to the game entry). And if you’re talking about the thing that is called “version” in the script edit form ­– again, it’s a freeform string, so whoever creates it can type in it whatever he likes (it’s essentially a tl;dr equivalent of description intended to tell the difference between install scripts of one game); so there’s gonna be different “version” strings for the same runner, as well as “versions” that are unique to a specific game (likely multiple for one game).

In the end, if your goal is to determine which platforms have installers, your best call would be to base it on keyworded data from those installers (like the Runner field).

That’s what bug reports are for :wink:

I’m same as you, just a guy who has nothing better to do than waste time writing stuff on a forum about trying to play games on Linux :smiley:

As for your question, they’re likely browsing the forum like most folks – when they have time, and looking at topics they want to look at. If you want to find them, check who closes issues on the bugtracker – the head dev uses a capybara head as avatar, I believe…

Contact – sure (I think this forum has a private message feature, for instance). Report a bug or request a feature? No, that’s what bugtrackers are for :slight_smile:

And don’t forget that to use their own bugtracker rather than owned by someone else they’d have to host it with their own money (as well as worry about stuff like DDoS), which is a bit much to ask from someone who does this as a hobby.

…Also, regarding IRC – I’m pretty sure the only requirement to enter the channel is to register your nickname on Freenode first.

I’m doubtful about that being a real policy because there are installers which don’t allow such freedom. Here is an example where the installer author is allowed to restrict the choice of the user: Installers may have URLs in the files: section, and those files will be pulled without the user being to override that, at least at install time (see for instance the installer for Flight Of The Amazon Queen: It simply downloads the game from somewhere, you get no option to use your own copy of the game. Precisely because it’s helpful). But obviously, you’re always allowed to change the files after installing.

Obviously, I can make it an option, it’s just a matter of coding that. But I would just like to have some confirmation from a dev it would actually be of interest before spending more time on that code.

In the meantime, I made another little enhancement: This patch makes the app remember the last directory selected between invocations of the installer, which can help when you’ve multiple games in the same directory, so you don’t have to browse from your $HOME to that directory each time you install a new game. Here’s a link to it in case anyone is interested.

Yes, I do get what you say. Sure, it’s just a text field. But if you browse NES games (for instance, most platforms are like this) you’ll see that pretty much all of them have an installer with a version string that just says NES, so the end result is that searching for games where version == NES will just bring them up, which would be useful, at least for me.

The goal is to find which games have an installer for a definite platform. Could you suggest what search parameters would yield something like that?

Also: What search could show me all the games (or installers) which have been added since a period of time (like, last month)?

Yep, just tried again, seems to have worked. Strange though: NickServ kept telling I was logged in but it didn’t work before. Go figure.

In case of this game, it’s freeware and that install script simply allows you to install the version downloadable from the Web without having any files. It does not by itself prevent the user from installing the game from any other file (for instance by using another install script); in fact, the script description directly informs users that this script is specifically made for that particular freely downloadable file.

As I mentioned, when generating/updating search index, it should be possible to calculate the platform of the install script accurately by its contents; for instance, if runner is set to libretro, and script.game.core is set to fceumm, it’s guaranteed to be a NES game regardless of what the user typed in the “version” string.
You might even convince the devs that displaying the platform tag on the script list item might be a good feature, while you’re at it (it might even make them more willing to spend the effort on implementing it in the first place).

…Are you asking me about already existing options? I really doubt that this kind of search will be available anytime soon, simply because of implementation effort involved. You’ll need a datetime input field (one that doesn’t suck and works in all browsers, an important distinction) on the frontend, additional query parameter(s), as well as search queries and search indices for those new parameters (that would be game addition timestamp – if they keep that, – as well as the date of last script update – you’ll have to check which scripts for the game were updated, rather than having them implement a new type of search altogether). Yes, it’s not impossible to implement, but unless it’s a highly requested feature, you likely won’t have it added within near future unless you make the pull-request yourself. (And convince the devs you’re not the only one who will use it.)

Yes, by using another install script. Which my little feature for the installer does not preclude at all, either. Anyway, let’s just agree to disagree and leave it at that, lest this devolves into one of those protracted internet arguments. Instead, let’s try some constructive discussion: Apart from that specific feature (which I’m not that bent upon, anyway), did you find any of the other features and fixes I posted useful? If not, what problems did you find (either functional, or code quality problems: I’m not a Python expert)?

Indeed, this is the support forum, so I’m asking for ways to get what I’m seeking from the web site (or the API, BTW, I can live making my search through the API if the frontend hasn’t got the features). But if you aren’t aware of any way to do that, that’s fine.

I haven’t really looked at your code, as I’m not really familiar with the codebase (and thus can’t tell what you’re breaking); I can only say that some of these features are kind of dubious (the slug one, for instance, might be unpopular with people who don’t actually see any need to install multiple copies of the same game – which I believe are majority). I’m also unsure of the point or use of this “directory” field (it should be present already for some games, and for others there’s not much difference between changing path to the game binary and the folder it’s in).

In case of incorrect scripts, if you’re actually detecting them correctly (as in, this isn’t the intended behaviour of a script that you’re misusing), the proper behaviour would likely be to inform the user that the script is outdated and should be changed by someone, rather than trying to guess what’s the author of the script intended for it to do (in general case, at least). Same probably goes for the Frotz games (a mistake in the script, which should be fixed in the script), but I’m not particularly sure about it.

Frotz is a runner, and replacing it with another runner by a checkbox isn’t really a good idea; I’d suggest to either add the runner, or whip up an “alternative binary” that mimics Frotz externally but internally uses your runner (and thus can be used with “custom path”). The latter seems preferable as the devs would probably not want to maintain two similar runners for the same format (not to mention needing two install scripts per game).
Also, I prefer Gargoyle :slight_smile:

In general, the normal strategy of “I made some fixes” would be to make pull-requests out of those patches, and discuss the usefulness/acceptableness of these changes with devs rather some random guy on the forum :smiley:

P.S. If you do that, also make sure to create a bug report for the runner game lists on the website project, didn’t notice those first time around. And a feature request for existing installer platform search, I guess.

You’re probably right. I’ll add a switch to enable it in system options.

This is the SQLite DB field directory, which is actually used in the code, for instance by the uninstaller, to check what folder it needs to remove. I prefer being able to change this from a field in the UI rather than edit the DB through SQL when I move a game around (granted, I don’t do it frequently).

True.

Actually, that’s what I did at first, but then I got tired to browse for the custom runner binary every time I switched for testing ;+)

That’s what I wanted to do, but pull requests also require people to open a GitHub account. Not worth the hassle for a few patches, I’ll just go play the games instead, I already spent too much time coding ;+) Thanks for your answers anyway.

Alright, exposing an existing field does make sense. Though I’m not sure about reusing it for other functionality.

if DEBUG: :smiley: