Page 1 of 2

PERFORMANCE IMPROVEMENT: TESTERS NEEDED! (was: UNIX sysadmin observations)

Posted: Fri 26 Apr 2019 08:13
by Atari_Datacenter
Hi, everyone. This is a great project, isn't it?

Me: I've been supporting Solaris servers for 20 years, but now I'm finally making the dive into Linux. The RetroStone is part of my at-home edutainment. I've only had it a full day, but I'm enjoying it and learning a ton about those Linux-specific quirks and also the H3 chipset. I'm complete inside AND outside of my comfort zone all at the same time.

Display devices: One of the things I've been playing with was the need to fully reboot the RetroStone to make a display change. I see that configuration changes are being done exclusively through the FEX script (converted into a binary) in the /boot folder. In Supreme RetroStone v1.3, those changes are facilitated by the video device selection script: /home/pi/RetrOrangePi/BashTool_ROPI_RCA/Tool/bashFiles/main.SH

That certainly works, but FEX was designed as boot-time hardware settings. The chipset wasn't designed to be locked into a single mode until the next time it is powered up. Has anyone tried changing display settings while the system was running by shifting the correct values into the H3's memory-mapped registers and reconfiguring/restarting any of the necessary demons? I've found a few web pages that won't solve the issue right out of the box, but they point to some of the components necessary to pull it off:

Dynamic video mode changer: https://forum.doozan.com/read.php?6,9002
mapping register with mmap in C (gpio in this example): http://docs.cubieboard.org/tutorials/co ... on_lubuntu
Working with multiple frame buffers: http://linux-sunxi.org/Dual_Monitor_Support
Informing the OS of the new resolutions with fbset: http://linux-sunxi.org/Display

Display options: Changing output devices and framebuffers might be a final goal. But a step along the way might be a video mode change, or working with a large virtual video buffer and seeing if the built-in scaler will give us better perceived resolution on the 320x240 display? But along the way, I've made a few attempts to far at video cloning (having the HDMI and the analog "CVBS" display the exact same output) but I haven't yet had a success. I am seeing that running two displays at the same time seems to significantly drive up the internal temperature of the RetroStone. It makes sense I guess. If you're taxing the CPU it is going to get hot, but if you start taxing the GPU it can quickly push the heat too high for the chipset.

One of the things I was really hoping to see is if we could get a game playing on the HDMI connection at the same time the local LCD has its own independent framebuffer that we can use to display some static text or graphics. But even two static screens and only moderate CPU use may end up tickle the heat issue and creating stability issues. Anyone have any experience playing with any of this?

SSH'ing into the RetroStone: I haven't heard many people talk about this, but it really is nice to be able to SSH into the RetroStone (via the "pi" account). And once you SSH into a server, the next step is to scp files back and forth with the RetroStone and a PC. It has been a really great way of loading ROMS onto the handheld and for backing up some configuration.

Deleting a theme can be dangerous: I had an experience where the RetroStone stopped working. It went to a text mode and display some assertion error about some sort of font texture cache. I looked in the forum and Googled and found that a number of people were running into similar problems now and then, but no great solutions I figured out how mine was happening, at least.

When you've got an author that is offering several themes, certain components (the fonts in particular) may end up being stored in the same place and shared among their different Themes. So while you've selected a Theme called "Mars" and you erase a totally different Theme that was installed earlier called "Venus", all of the sudden, emulationstation can't run and all you see is a font error that you can't get around! (And you probably didn't even realize those Themes had the same author, much less shared files.) The solution would be to log into the UNIX shell and remove/reinstall your active Theme, or to reinstall the Theme you just deleted. SSH is a good way to do it. When you log into as the "pi" user, these commands will start you down the road:
cd /home/pi/RetroPie-Setup
sudo ./retropie_setup.sh

So that's my experience as of day one.

Software-based LCD backlight power switch: Something I'd like to do is to tie into one of the GPIO pins and see if I can wire a simple circuit that would power on / power off the LCD backlight through some simple GPIO commands in the shell. I just find it annoying that the screen displays garbage while I'm playing games on the big screen, you know? Anyone have some thoughts on where I might want to focus my attention on the PCB?

The video conversion chip? Anyone have any details on how the CVBS (composite video) output from the A3 chipset is being massaged before sending it to the LCD? I'm wondering if it can be bent to deliver something close to full NTSC to a different LCD, or if the only real option is going to be to either tie into the composite video signal (or hijack the external HDMI) and convert and display it from there? I see that someone else here in the forum has gone the second route (internal HDMI display) but I'm hoping that there is some chance that the internal composite video may yet be salvaged?

Thank you for your time and interest,
Atari Datacenter

Re: Some observations from a UNIX sysadmin

Posted: Sat 27 Apr 2019 02:31
by deerwings
Heat is definitely an issue. I haven't looked at it yet to see what it could take to improve the cooling, but near as I can figure the best solution would be a larger heat spreader with an active cooling solution to encourage airflow out of the case. That would probably resolve the heat issue, though I'm not certain the efficiency versus the payoff. Worth exploring.

Regarding the screen: If running both internal and external screens at the same time drives the heat up that significantly, then it might be something that would only be practical with a sufficiently efficient cooling system but I don't know many other practical applications for it, though I can see where you're going with that.

Regarding the Themes: I've had this issue myself, and the best solution I have come up with is backing up your themes to the rom folder, and then if you make a mistake, you can simply copy it back. Otherwise, you'd probably be best off manually selecting a theme in the .emulationstation/es_settings.cfg to something else and then performing a system reboot. I've been personally using the Pixel-TFT with sufficient font settings to make it practical on the internal display.

I would like to see a better method to switch internal and external screens, though!

Re: Some observations from a UNIX sysadmin

Posted: Sun 28 Apr 2019 08:17
by Atari_Datacenter
Good news?

I worked on some of the performance problems with a game in MAME that I was quite familiar with: Tokimeki Memorial Taisen Puzzle-Dama. Sure, that's a pretty strange title forr an American to be familiar with, but I've got the game in my PCB closet as part of my Konami GX motherboard collection. On the RetroStone (with RetrOrangePi) the game had some problems. The graphics were faithful, but the the sound was scratchy at times, there were some subtle slowdowns from time to time for no apparent reason, and a regular audio click every two seconds or so. Playable, but annoying, right?

I think I've made some significant improvements in performance tuning the platform. I've been writing a detailed explanation of what I tried, what I found, and how I addressed it. At this point, it is 1am and I'm going to call it a night. I'll post what I found (and I guess probably sign up to the RetrOrangePi forum and lelt them know over there -- these are all OS changes, no hardware hacks).

Are there going to be any volunteers with shell access to their RetroStone to try a few commands and evaluate the change in performance?

Atari Datacenter

Re: Some observations from a UNIX sysadmin

Posted: Mon 29 Apr 2019 00:50
by Atari_Datacenter
Performance tuning UNIX servers is something I do in my day job, so even though I'm just now getting up-to-speed with Linux, I thought I'd try my hand with some tuning of the RetrOrangePi distribution on my newly acquired RetroStone. At a high level, my findings (if accurate) seem to suggest that the RetrOrangePi should be customized to match the intended form factor (laptop, gameboy form-factor handheld), hardware (desktop, RetroStone), and intended use (entertainment/emulation, or desktop). Those customizations could be a one-time adjustment, or settings that dynamically changed based on which function the user has launched. But I'm getting ahead of things.

I NEED VOLUNTEERS WHO WANT TO IMPROVE THEIR RETROSTONE'S PERFORMANCE.
IS THAT YOU?

I think we'd all benefit if some others would replicate the change with the steps listed below. Then let us know if you think the benefits are real, and perhaps you could report back with your own before/after story. No permanent changes will be made to your OS image or your hardware settings. This requires root access to the device while a game is actively running on the screen (ssh would make a fine choice).

HOW TO MAKE THE PERFORMANCE CHANGES AND PROVIDE TESTING FEEDBACK:

STEP 0 (BASELINE): Use only the internal LCD display. Reboot your RetroStone. If you can report what frequency scaling governor your CPUs are set to, that's appreciated. (I used "cpufreq-info" as part of the "cpufrequtils" package.) And if you can report your CPU and DRAM frequencies ("h3consumption -p"), that too is helpful. I've been using the arcade game "Tokimeki Memorial Taizen Puzzledama" because I have it in my arcade pcb collection and I know exactly how it behaves. If you can do that game, great. If you have another game that gives you some mild performance issues, you can try and report on that instead. Whatever game you've selected, please launch it before proceeding to the next step.

STEP 1 (ENFORCED CPU SETS): We're going to specifically require that certain items run on pre-designated CPUs, and at very specific priority levels. With the game still running, as root, cut and paste the following commands into your RetroStone (and in the order they're listed). The first line binds everything (that can be reassigned) to run only on CPU 0. The for loop which follows will bind all retroarch threads to the scheduler's choice of either CPU 0 or CPU 1, and with a extremely high priority level that beats almost anything else. The next two lines will re-bind the main retroarch thread (the one that consumes an entire CPU) to CPU 2 where it will run all by itself. The final two lines will re-bind the SDLAudio thread (presumably responsible for audio) to CPU 3 where it will run all by itself. This probably isn't the ideal set of bindings, but in the absence of certain details, it still makes for a good improvement. NOTE: If you exit a game and start another, run these commands again after the emulator has loaded the new game. When you are done testing, reboot your RetroStone to remove the bindings, or everything may actually run slower until rebooted.

Code: Select all

for i in `ps -eao pid`; do echo $i; taskset -pc 0 $i; done
for i in `ps -eLo tid,args | grep retroarch | grep -v grep | cut -c1-6`
do
chrt -r -v -p 90 $i
taskset -pc 0,1 $i
/usr/bin/renice -n -1 -p $1
done
taskset -pc 2 `ps -eao pid,args | grep bin/retroarch | grep -v grep | cut -c1-6`
chrt -f -p 10 `ps -eao pid,args | grep bin/retroarch | grep -v grep | cut -c1-6`
taskset -pc 3 `ps -eLo tid,fname,args | grep retroarch | grep SDLAudio | cut -c1-6`
chrt -f -p 10 `ps -eLo tid,fname,args | grep retroarch | grep SDLAudio | cut -c1-6`
STEP 2 (DISABLING UNNEEDED SERVICES): I thought the above tuning helped, but I was still getting regular clicks or semi-slowdowns every couple of seconds. I looked for the potential reasons and started to install strace when I found two Linux processes which were regularly popping up. (Neither one is necessary for a RetroStone.) The following commands will disable the services. Unless you've rebooted, you only need to run this once. These settings revert back to normal on the next reboot.

Code: Select all

systemctl stop acpid.path acpid.service acpid.socket
systemctl stop systemd-journald.service systemd-journald.socket systemd-journald-dev-log.socket
STEP 3 (TESTING): To make my own judgment on performance quality with my own choice of Tokimeki Memorial, I used the attract screen (which had sound) and then gameplay. The sound was significantly better and those slight slowdowns that were happening at regular intervals were no longer a problem. If you choose a different game, that's fine. I've targeted these tunings for the Arcade game selections (using MAME) but I *believe* they'll also work on some of the other emulators.

STEP 4 (REPORTING BACK): All done? Report back and let everyone know how it went with a reply to this topic. Any time someone can run a performance test on real machines, we all benefit. Thank you!

So, reboot your RetroStone, run the tests, and let us know how things go, willyah?

Thanks,
Atari Datacenter

Re: Some observations from a UNIX sysadmin

Posted: Fri 3 May 2019 15:48
by Admin
Hi! Sorry I missed your message!

About HDMI switch I tried for a long time to find a way to make automatic switch between LCD and HDMI rather than using the bashfile but with no luck.
You can see discussion here on armbian forum : https://forum.armbian.com/topic/5870-av ... ment-64676

In the end I could read HDMI detect (HPD) but after impossible to change the display. Turning OFF LCD is OK, turning HDMI ON is another matter. If you want to give it a shot :)

From what I read on armbian forum, for H3 dual output of composite and HDMI with 2 different framebuffer is not working currently. I made some tests without success. Maybe clone '2 screen one framebuffer) could work but I could not get it to work.

SSHing is working.

Software-based LCD backlight power switch: The backlight itself is driven by AMT630A which is the RGB-CVBS converter. You can't enable/disable the power supply of the backlight directly from software. In fact it should be possible as when you turn OFF the device from software the LCD ends up being turned off. I think there's probably a signal to send through CVBS that tells AMT630A to power off. Maybe it's just when CVBS is flat 0V?

Video conversion chip is AMT630A it converts composite into RGB24. I can send you datasheet if you want.

Re: Some observations from a UNIX sysadmin

Posted: Sat 25 May 2019 03:26
by Atari_Datacenter
Thank you for all of those answers. I'll accept your offer of the AMT630A datasheet. That'll be a great help! What I'd like to see is if I can do a logical AND (in hardware) between the PWM output(s) of the AMT630A and the output of one of the unused GPIO pins. At a high level, that would allow me to turn off/on backlight with a binary output to that GPIO pin. While I wait on that datasheet, something else has caught my attention:

Now that I've played around with the RetroStone a little bit more, I've come to believe that the steps in my previous message didn't solve the performance issue that I'm seeing. What issue, you ask? There seems to be a very regular slowdown at regular cycles which are something like 2.0 or 2.5 seconds apart. I see it when I'm doing something as simple as moving up/down on an ordinary list of arcade ROMS, or something complex like inside of a game and there is a cyclical slowdown in the sound or action.

When I first investigated it, some unnecessary ACPI and journal related system services (and worker processes) caught my eye, so I included the instructions to disable them. After that, there were no more processes that were getting on the CPU that weren't either critical to the OS running or the emulator running.

But also around those slowdowns, I started seeing some kernel threads starting and stopping. It could be nothing, or it could be the underlying cause. Two examples might be: general memory management routines (putting clean pages back on the free list), flushing I/O writes (moving dirty pages back to disk). My next step is to either go with systemtap or ftrace to figure out what's going on inside the kernel and see if we can nail down those latencies.

Crossing my fingers,
Atari Datacenter

Re: Some observations from a UNIX sysadmin

Posted: Sun 26 May 2019 06:03
by Atari_Datacenter
MY GOD, I FIGURED OUT WHERE THE SLOWDOWN IS COMING FROM.

I started to bring in some tools like ftrace and others to help examine where things were going wrong. After installing one tool and looking at page faults, I noticed that systemd-udevd was going through a regular cycle of activity. And the duration of a cycle seemed to match my slowdown. After disabling it, you know what happened? My micro-pause every 2.0 to 2.5 seconds completely disappeared. My games that had a mild slowdown were now running at full speed. I'm highly confident that systemd-udevd is the cause of many of our performance problems.

I've already put together some code that can safely demonstrates that if this issue is addressed, most performance issues will go away. It doesn't seem like many people are active here on this forum anymore. Are there still some people left who understand the slowdowns that I'm talking about and have remote shell access into their Retrostones to try the proof-of-concept? Should I start another thread?

Thanks,
The Atari Datacenter Man

Re: Some observations from a UNIX sysadmin

Posted: Sun 26 May 2019 11:58
by retriok
I am interested in your work, unfortunately my knowledge is not enough to be of help. why don't you ask for help in the forums of retroange pi and especially alexkidd? there you will surely find someone who can help you.

Re: Some observations from a UNIX sysadmin

Posted: Sun 26 May 2019 19:22
by deerwings
I would be willing to try! I haven't noticed any major slowdowns but I could see some room for improvement. Just let me know what you need me to try and I'll see what I can do.

I've been fighting personally with trying to hide the console screen during launch of an emulator, but it seems like this is something that's hardcoded into Emulationstation and Raspbian but I have gotten pretty close.

Re: Some observations from a UNIX sysadmin

Posted: Tue 28 May 2019 02:48
by Atari_Datacenter
I'm going to provide a test plan. This is not intended to be a long-term solution. It is only to test to see if we've identified a performance problem and that we addressed it in a way that improves performance.

SUMMARY: You'll run through a formal test plan that provides some structure for your own testing. It will be up to you to identify what you are going to test, how you will test it, and how you will grade the results. We'll measure your existing system, disable some suspected services, measure again, re-enable those services, and then test again and make sure that everything is back to its original condition. In between all of the major steps, we are rebooting the Retrostone in order to ensure that we have a clean environment for our testing and to quickly catch any abnormalities during out testing.

PREFERRED: Having remote shell access into your Retrostone (such as SSH). Sure, you could just plug in a keyboard, but you may find it difficult to type in those scripts exactly as they are listed.

REQUIRED: The ability to run commands AS ROOT. (You should be able to run any command as root by first typing "sudo /bin/sh", then the command, then "exit" to return back to your normal account. Do not simply put "sudo" in front of the provided scripts. It won't work.)

RISKS: If done correctly, very low. You're only shutting down services in systemd and re-enabling them when you're done testing.


YOUR TESTING INSTRUCTIONS ARE BELOW:

Every single step is important. If you do not understand a step, seek assistance.
Never skip a step unless instructed!


1. REBOOT YOUR RETROSTONE: We need to make sure that we start our testing with a fresh system.

2. OBTAIN A BASELINE: Determine exactly which games you're going to test with, how they behave at certain points, what you're testing for, and how you grade those results.

3. DISABLE SUSPECTED SERVICES: Be careful! Copy the script by clicking on SELECT ALL (below) and then (for Windows) Control-C to copy the selected text . As the root user, paste the command into a root shell prompt on your Retrostone. Hit ENTER a single time to run the script.

Code: Select all

for i in systemd-journald.socket systemd-journald-dev-log.socket acpid.socket acpid.path systemd-udevd-control.socket systemd-udevd-kernel.socket systemd-udev-settle.service systemd-udev-trigger.service systemd-udevd.service udev-finish.service udev.service acpid.service systemd-journald.service; do echo $i; systemctl mask $i ; systemctl disable $i ; systemctl stop $i; done
4. FREE PARKING: This step was counter-productive and so it was removed. Proceed directly to your testing in Step #5!

5. TEST THE CHANGE: Again play those games that you identified in your baseline and observe how those key aspects have or have not changed. Do your evaluations and scoring that you set up in Step #2.

6. RESTORE SUSPECTED SERVICES: Be careful! Copy the script by clicking on SELECT ALL (below) and then (for Windows) Control-C to copy the selected t ext. As the root user, paste the command into a root shell prompt on your Retrostone. Hit ENTER a single time to run the script.

Code: Select all

for i in systemd-journald.socket systemd-journald-dev-log.socket acpid.socket acpid.path systemd-udevd-control.socket systemd-udevd-kernel.socket systemd-udev-settle.service systemd-udev-trigger.service systemd-udevd.service udev-finish.service udev.service acpid.service systemd-journald.service; do echo $i; systemctl unmask $i ; systemctl enable $i ; systemctl start $i; done
7. REBOOT YOUR RETROSTONE: We'll want to make sure that it reboots okay and everything is back to normal.

8. TEST THE CHANGE AGAIN: Did it reboot correctly? If you identified something that ran slower or faster, try it again. Is it back to its original condition? Or is it somehow faster or slower? (This is a repeat of step #5, but this time you only need to do a very casual and quick observation.)

9. REPORT BACK: Reply in this thread and let us know, right? These commands are just a workaround. If they're proven to improve or even eliminate certain performance issues, we can implement a more permanent solution.

NOTE: If you are using something other than the built-in controls, they may become unavailable during testing. If so, just for testing purposes, use the built-in buttons. Dont worry: the eventual solution shouldn't need you to do that.

QUESTION: "I want to try it out, but this is too much. Can you make it simpler?"

Yes! At a minimum, you'd want to do steps #3, #5, #6, #7. I'd still encourage you to do step #9 and let us know how it went. Sure, you'd be letting us know that you did a quick informal review, but that kind of experience is still useful and appreciated!