Olivier Langlois's blog

Update 4/06/13:

Cool, A bug fix to this discovered bug has been integrated into Vi0L0 ArchLinux catalyst package:

( 13/106) upgrading catalyst-utils [#####################################] 100%
---------------- I/O BUG -----------------------------------------
There's a bug in fglrx found by lano1106 which generates
great amount of unneeded I/O operations

To activate workaround enable systemd service:
systemctl enable temp-links-catalyst
systemctl start temp-links-catalyst

More infos:
https://bbs.archlinux.org/viewtopic.php?pid=1279977#p1279977
https://bbs.archlinux.org/viewtopic.php?pid=1280193#p1280193
----------------------------------------------------------------

All I did during the strace session is calling glxinfo.

ano1106@whippet2 ~ $ su
Password:
whippet2 /home/lano1106 # ps -ef | grep X
lano1106 640 617 0 22:28 tty1 00:00:00 xinit /etc/xdg/xfce4/xinitrc -- /etc/X11/xinit/xserverrc
root 641 640 0 22:28 ? 00:00:02 /usr/bin/X -nolisten tcp :0 vt1
root 2965 2957 0 23:31 pts/2 00:00:00 grep --color=auto X
whippet2 /home/lano1106 # strace -p641 -o/tmp/x_strace2.txt
Process 641 attached
^CProcess 641 detached
whippet2 /home/lano1106 # grep amdpcsdb /tmp/x_strace2.txt
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
whippet2 /home/lano1106 # grep atiapfxx.blb /tmp/x_strace2.txt
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35

amdpcsdb is about 30 KB long. Considering that glxinfo execution is slightly less than a second long and during that time the file recreated 30 times, that rougly means 1 MB/sec writing on disk if you invoke short opengl applications in a tight loop. It seems to happen on startup of any mesa application.

Say goodbye to your SSD lifespan :-)

Here is what I have done:

1. Enable crossfire
2. Start x
3. start oprofile
4. execute glxinfo
5. Boom!
6. collect oprofile data

I have slightly reformated oprofile output to just keep # of calls and function name:

1438298 OS::readPort(Asic*, unsigned int)
47365 KCL_GlobalKernelScheduler
41736 Asic::WaitForBitsClear::ConditionSuccessful()
25689 Asic::WaitUntil::WaitForComplete()
7253 KAS_GetTickCounter
2 KCL_SPINLOCK_STATIC_Grab
1 KCL_SEMAPHORE_STATIC_Down

After having waited several minutes, I have redone a second oprofile session:

256155 KCL_GlobalKernelScheduler
31828 firegl_hardwareHangRecovery

Note that there is no corresponding spinlock unlock or semaphore_up. I am not totally sure that oprofile is 100% accurate or if it does some kind of statistical profiling where you do not get a 100% accurate picture but since X becomes hang and unkillable, what oprofile shows seems accurate. fglrx is stuck in a deadlock.

I will redo the experiment without Crossfire to see what is the typical fglrx profile when invoking glxinfo.

No idea what is the value added of this support.

I have modified the function KCL_EFI_IS_ENABLED() to return 0 on my UEFI enabled system and it made 0 difference.

It will remain like that until I have an explanation. In my experience, UEFI usage is rarely benefiting the end-user.

I have recompiled my kernel with CONFIG_AGP=n (The last chipset with AGP support is at least 10 years old and only Intel integrated graphics (like i915) still use the AGP code as a hack) and by doing so, it has broken fglrx with:

lano1106@whippet2 ~ sad $ dmesg | grep fglrx
[ 2.897539] fglrx: module license 'Proprietary. (C) 2002 - ATI Technologies, Starnberg, GERMANY' taints kernel.
[ 2.899186] fglrx: Unknown symbol KCL_AGP_FindCapsRegisters (err 0)

I have just moved this function outside the #if CONFIG_AGP block and it is ok to do so since it does not contain anything specific to agp. only generic pci functions.

You can get the patch at:

http://www.trillion01.com/patches/kcl_agp_13_4.patch

Here is what I have done:

1. Enable crossfire
2. Start x
3. Start tracing X with strace
4. execute glxinfo
5. Boom!

The last entry in my trace log is:

ioctl(13, 0xc03064a6

where 13 is a file descriptor to fglrx fs frontend.

lrwx------ 1 root root 64 May 24 23:36 13 -> /dev/ati/card0
lrwx------ 1 root root 64 May 24 23:36 14 -> /dev/ati/card1
lrwx------ 1 root root 64 May 24 23:36 15 -> /dev/ati/card0
lrwx------ 1 root root 64 May 24 23:36 16 -> /dev/ati/card0
lrwx------ 1 root root 64 May 24 23:36 17 -> /dev/ati/card0

Next step will be activate kernel profiling, with some luck, maybe I'll find out that this is a simple to fix deadlock in spinlocks located on the public side of the driver.

In the meantime, this exercise did allow me to find out that the catalyst X driver was doing silly things. Such as totally rewrite the amdpcsdb file several times in the span of about a minute:

lano1106@Wailaba2 /dev/snd $ grep amdpcsdb /nas/x_trace.out
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33

and reading a 3 bytes file containing " \n" 5 times:

lano1106@whippet2 /etc/ati $ ls -l atiapfxx.blb
-rw-r--r-- 1 root root 3 Apr 27 15:35 atiapfxx.blb
lano1106@whippet2 /etc/ati $ grep 'atiapfxx.blb' /nas/x_trace.out
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33

I have sorted out the IOMMU thing.

#if defined(CONFIG_AMD_IOMMU) || defined(CONFIG_DMAR)
#define FIREGL_DMA_REMAPPING
#endif

I have noticed CONFIG_DMAR does not exist anymore. It has been replaced by CONFIG_INTEL_IOMMU.

For people having a Intel chipset with VT-d. It might be worthwile to try. To enable FIREGL_DMA_REMAPPING,

1. replace CONFIG_DMAR by CONFIG_INTEL_IOMMU in firegl_public.c and rebuild the driver
2. Enable VT-d in BIOS
3. Edit kernel command line param in grub.cfg by adding 'intel_iommu=on'

The whole purpose FIREGL_DMA_REMAPPING is to avoid compatibility issues between different kernel version in regards to DMA API. On most distribution CONFIG_AMD_IOMMU and/or CONFIG_INTEL_IOMMU will be defined and when fglrx will be compiled, FIREGL_DMA_REMAPPING will be defined even if at runtime no HW IOMMU will be detected and it works fine anyway.

DMA API is fine. Interface is the same for HW and SW IOMMU.

DMA api ends up using functions in arch/x86/kernel/pci-nommu.c if no hardware IOMMU is on the system. So, unless proven wrong, I would say force FIREGL_DMA_REMAPPING for all setup.

That being said and after having done a fantastic journey in IOMMU and DMA kernel code, I have found out that my problem was much simpler than that:

In /etc/X11/xorg.conf, section "ServerLayout"

2 screens, each one associated to one of my 2 cards, were enumerated. I have removed the second screen of the section and by doing so, it has allowed me to activate Crossfire.

Now, X server starts but hard freeze as soon as I try to access opengl functionality. It can be something as simple as 'glxinfo'.

lano1106@whippet2 ~ $ aticonfig --adapter=0 --crossfire=on
Crossfire chain(s) enabled
CrossFire does not support on this platform
Warning: X needs to be restarted before CrossFire changes take effect.


When you get the message "CrossFire does not support on this platform", CrossFire is not enabled. I wish the error would be more verbose than that but I think that I have good lead to investigate:

lano1106@whippet2 ~ $ lsmod | grep fglrx
Module Size Used by
fglrx 5176689 203
amd_iommu_v2 7252 1 fglrx
button 4669 1 fglrx


The interesting info here is that amd_iommu_v2 is used by fglrx and


lano1106@whippet2 ~ $ dmesg | grep IOMMU
[ 2.614901] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[ 2.614902] AMD IOMMUv2 functionality not available on this system


I'll investigate the situation further but it makes sense that the AMD IOMMU does not work on my Intel board (ASUS P9X79 WS). The concept of IOMMU is totally new to me and still a bit fuzzy but apparently is provided by Intel through VT-d that you normally enable by:

1. Turn it on from the bios
2. Add the kernel param intel_iommu=on

and I am glad to say that I modestly contributed to its enhancement.

BFS author, Con Kolivas, has been kind enough mention my small contribution in the release entry on his blog:

http://ck-hack.blogspot.ca/2013/05/bfs-0430-ck1-for-linux-39x.html