I always had the intuition that allocating and initializing the memory to 0 in a single step could be faster than doing it yourself even if modern C compilers replace the memset() calls with inline assembly instructions. By browsing the glibc malloc source code for another problem, I had the perfect opportunity to validate my intuition and it turns out that my intuition was correct!
On Linux, the glibc heap manager is using the sbrk() system call to grow the heap. The fresh memory returned by sbrk() is initialized to, guess which value??, 0. glibc heap manager keeps track of memory in its heap that is freshly returned by sbrk() and calloc() can leverage this information to return memory already containing zeros and skipping totally the memset() step!
Update:
I did benchmark what I was writing and I did find out that, with glibc 2.18, it is true only for allocation size > ~ 64KB. Smaller than that malloc + memset is faster. I have reported this finding to the glibc mailing list. I still advocate to prefer calloc() as the current result is probably temporary and I expect future version to remove this anomaly.
for those desiring to play with the test, you can get these files:
to build do:
gcc -O3 -c calloc_emul.c
Play with the various define values and
gcc -O3 -c tst-calloc.c
gcc -o tst-calloc tst-calloc.o calloc_emul.o
When I start an application with glc-capture, the application become frozen and unresponsive.
It even ignores kill -11 signals!
With gdb, I have found that glc blocks inside a dlopen() call.
(gdb) where
#0 0xf76ff430 in __kernel_vsyscall ()
#1 0xf7591231 in __lll_lock_wait_private () from /usr/lib32/libc.so.6
#2 0xf750fd8a in _L_lock_6856 () from /usr/lib32/libc.so.6
#3 0xf750d77d in malloc () from /usr/lib32/libc.so.6
#4 0xf770d05b in _dl_map_object_deps () from /lib/ld-linux.so.2
#5 0xf7712f8b in dl_open_worker () from /lib/ld-linux.so.2
#6 0xf770ee1a in _dl_catch_error () from /lib/ld-linux.so.2
#7 0xf7712954 in _dl_open () from /lib/ld-linux.so.2
#8 0xf75bb67b in do_dlopen () from /usr/lib32/libc.so.6
#9 0xf770ee1a in _dl_catch_error () from /lib/ld-linux.so.2
#10 0xf75bb76b in dlerror_run () from /usr/lib32/libc.so.6
#11 0xf75bb7f1 in __libc_dlopen_mode () from /usr/lib32/libc.so.6
#12 0xf7591ad8 in init () from /usr/lib32/libc.so.6
#13 0xf769c0ee in pthread_once () from /usr/lib32/libpthread.so.0
#14 0xf7591d45 in backtrace () from /usr/lib32/libc.so.6
#15 0xf74adc63 in backtrace_and_maps () from /usr/lib32/libc.so.6
#16 0xf7504263 in __libc_message () from /usr/lib32/libc.so.6
#17 0xf750a3ca in malloc_printerr () from /usr/lib32/libc.so.6
#18 0xf750be11 in _int_malloc () from /usr/lib32/libc.so.6
#19 0xf750d788 in malloc () from /usr/lib32/libc.so.6
#20 0xf770d05b in _dl_map_object_deps () from /lib/ld-linux.so.2
#21 0xf7712f8b in dl_open_worker () from /lib/ld-linux.so.2
#22 0xf770ee1a in _dl_catch_error () from /lib/ld-linux.so.2
#23 0xf7712954 in _dl_open () from /lib/ld-linux.so.2
#24 0xf768bcbc in ?? () from /usr/lib32/libdl.so.2
#25 0xf770ee1a in _dl_catch_error () from /lib/ld-linux.so.2
#26 0xf768c37c in ?? () from /usr/lib32/libdl.so.2
#27 0xf768bd71 in dlopen () from /usr/lib32/libdl.so.2
#28 0xf76c6077 in get_real_alsa () at /home/lano1106/dev/lib32-glc/src/glc/src/hook/alsa.c:256
#29 0xf76c64d9 in alsa_init (glc=glc@entry=0xf76cd280 <mpriv>) at /home/lano1106/dev/lib32-glc/src/glc/src/hook/alsa.c:96
#30 0xf76c29ab in init_glc () at /home/lano1106/dev/lib32-glc/src/glc/src/hook/main.c:114
#31 0xf76c56f5 in __opengl_glXGetProcAddressARB (proc_name=0x928a8c2 "glDeleteFencesNV") at /home/lano1106/dev/lib32-glc/src/glc/src/hook/opengl.c:332
#32 0xf76ee80c in glXGetProcAddressARB () from /home/lano1106/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so
I was suspicious about malloc_printerr() call so I went to glibc malloc source code. BTW, I have found something else interesting to share in my next post concerning glibc heap management. My suspicion was right. When you see malloc_printerr() in a stack, it is pretty much game over as it means that malloc() has detected memory corruption in the heap.
Since it is happening even before main() invoked when shared libraries are loaded, even __libc_message() needs something to be loaded by the dynamic loader which will reenter malloc and auto self deadlock itself!
Fortunately, I am entering this state only when using a particular glc command line option, hence it should not be too hard to find the offending code.
Update: I have found the bug!
https://github.com/nullkey/glc/pull/62
I was looking to screen capture opengl gaming sessions on Linux. On my first attempt, I have tried to use ffmpeg x11grab. This initial attempt failed since all recent OpenGL graphic cards drivers support direct rendering. This means OpenGL rendering simply bypass communicating with the xserver and makes x11grab useless. x11grab will grab anything on the screen that is behing the fullscreen foreground opengl window that use direct rendering.
I have then found glc that even if there is no recent activity on its github page, is very effective in what it is doing. It has not a lot of documentation on it but you can find a basic usage wiki page on it.
How does glc works?
It somehow place itself between the application to capture and the ALSA and OpenGL APIs with the help of a library called elfhacks, intercept the raw data and store it in a glc file that can become huge. The fact that it only support only very minimal realtime compression is one of the weakness of the tool. To give you an idea on how big the file can become, here is a quick calculation for a 1920x1080 resolution: 1920x1080 is about 2 millions pixels. Each pixel taking 4 bytes, it gives about 8MB per frame. At 30 fps, it is little bit less than 250MB per second! If you apply compression, I think that I have seen disk space consumption close to 1GB/minute. I have the plan to modify glc to pipe the captured raw data to ffmpeg for possibly improve a little bit the compression.
I intend to write a couple of posts on glc because there is not a lot written on it and hopefully it will help many to start using it so I'll cover things that I had to set up to be able to use it. The first thing will be to clarify glc audio capture options. I was under the impression that I had to use -a option and to provide an ALSA capture device to capture the application pcm streams. glc can record several pcm streams.
First it places ALSA hooks to intercept the application calls to ALSA to record the applications pcm stream. This is the default behavior and nothing needs to be added to the command line to record this audio. You can disable this recording with the option --disable-audio.
Secondly, you can add additionnal streams by using the -a option.
There exist a third option that I have developped due to my misinterpretation of how glc capture audio. You can disable glc ALSA hook recording and instead use ALSA snd_aloop driver fed by a pcm plug that split the stream between the sound card and the loopback device. This can be handy if you have surround 7.1 system with sampling rate of 192kHz and you want to downsample the stream and reduce the number of recorded channels. Even if this is not something you are confronted with, I will try to make it useful as an ALSA tutorial. ALSA lib doxygen output documents well the API but IMHO, it lacks some good examples to help understanding how to use ALSA.
On the GL side, frames can be acquired by 2 different methods: glReadPixels or GL_ARB_pixel_buffer_object (PBOs). You can use the switch --pbo to indicate to glc to try to use PBOs for the frame acquisition. PBOs have been introduced after glReadPixels. Their usage can provide a performance hedge over the former method for reasons that escape my undertanding. I will try to benchmark both methods to be able to compare them.
Update 4/06/13:
Cool, A bug fix to this discovered bug has been integrated into Vi0L0 ArchLinux catalyst package:
( 13/106) upgrading catalyst-utils [#####################################] 100%
---------------- I/O BUG -----------------------------------------
There's a bug in fglrx found by lano1106 which generates
great amount of unneeded I/O operations
To activate workaround enable systemd service:
systemctl enable temp-links-catalyst
systemctl start temp-links-catalyst
More infos:
https://bbs.archlinux.org/viewtopic.php?pid=1279977#p1279977
https://bbs.archlinux.org/viewtopic.php?pid=1280193#p1280193
----------------------------------------------------------------
All I did during the strace session is calling glxinfo.
ano1106@whippet2 ~ $ su
Password:
whippet2 /home/lano1106 # ps -ef | grep X
lano1106 640 617 0 22:28 tty1 00:00:00 xinit /etc/xdg/xfce4/xinitrc -- /etc/X11/xinit/xserverrc
root 641 640 0 22:28 ? 00:00:02 /usr/bin/X -nolisten tcp :0 vt1
root 2965 2957 0 23:31 pts/2 00:00:00 grep --color=auto X
whippet2 /home/lano1106 # strace -p641 -o/tmp/x_strace2.txt
Process 641 attached
^CProcess 641 detached
whippet2 /home/lano1106 # grep amdpcsdb /tmp/x_strace2.txt
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 35
whippet2 /home/lano1106 # grep atiapfxx.blb /tmp/x_strace2.txt
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 35
amdpcsdb is about 30 KB long. Considering that glxinfo execution is slightly less than a second long and during that time the file recreated 30 times, that rougly means 1 MB/sec writing on disk if you invoke short opengl applications in a tight loop. It seems to happen on startup of any mesa application.
Say goodbye to your SSD lifespan :-)
Here is what I have done:
I have slightly reformated oprofile output to just keep # of calls and function name:
1438298 OS::readPort(Asic*, unsigned int)
47365 KCL_GlobalKernelScheduler
41736 Asic::WaitForBitsClear::ConditionSuccessful()
25689 Asic::WaitUntil::WaitForComplete()
7253 KAS_GetTickCounter
2 KCL_SPINLOCK_STATIC_Grab
1 KCL_SEMAPHORE_STATIC_Down
After having waited several minutes, I have redone a second oprofile session:
256155 KCL_GlobalKernelScheduler
31828 firegl_hardwareHangRecovery
Note that there is no corresponding spinlock unlock or semaphore_up. I am not totally sure that oprofile is 100% accurate or if it does some kind of statistical profiling where you do not get a 100% accurate picture but since X becomes hang and unkillable, what oprofile shows seems accurate. fglrx is stuck in a deadlock.
I will redo the experiment without Crossfire to see what is the typical fglrx profile when invoking glxinfo.
No idea what is the value added of this support.
I have modified the function KCL_EFI_IS_ENABLED() to return 0 on my UEFI enabled system and it made 0 difference.
It will remain like that until I have an explanation. In my experience, UEFI usage is rarely benefiting the end-user.
I have recompiled my kernel with CONFIG_AGP=n (The last chipset with AGP support is at least 10 years old and only Intel integrated graphics (like i915) still use the AGP code as a hack) and by doing so, it has broken fglrx with:
lano1106@whippet2 ~ sad $ dmesg | grep fglrx
[ 2.897539] fglrx: module license 'Proprietary. (C) 2002 - ATI Technologies, Starnberg, GERMANY' taints kernel.
[ 2.899186] fglrx: Unknown symbol KCL_AGP_FindCapsRegisters (err 0)
I have just moved this function outside the #if CONFIG_AGP block and it is ok to do so since it does not contain anything specific to agp. only generic pci functions.
You can get the patch at:
Here is what I have done:
The last entry in my trace log is:
ioctl(13, 0xc03064a6
where 13 is a file descriptor to fglrx fs frontend.
lrwx------ 1 root root 64 May 24 23:36 13 -> /dev/ati/card0
lrwx------ 1 root root 64 May 24 23:36 14 -> /dev/ati/card1
lrwx------ 1 root root 64 May 24 23:36 15 -> /dev/ati/card0
lrwx------ 1 root root 64 May 24 23:36 16 -> /dev/ati/card0
lrwx------ 1 root root 64 May 24 23:36 17 -> /dev/ati/card0
Next step will be activate kernel profiling, with some luck, maybe I'll find out that this is a simple to fix deadlock in spinlocks located on the public side of the driver.
In the meantime, this exercise did allow me to find out that the catalyst X driver was doing silly things. Such as totally rewrite the amdpcsdb file several times in the span of about a minute:
lano1106@Wailaba2 /dev/snd $ grep amdpcsdb /nas/x_trace.out
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
open("/etc/ati/amdpcsdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
and reading a 3 bytes file containing " \n" 5 times:
lano1106@whippet2 /etc/ati $ ls -l atiapfxx.blb
-rw-r--r-- 1 root root 3 Apr 27 15:35 atiapfxx.blb
lano1106@whippet2 /etc/ati $ grep 'atiapfxx.blb' /nas/x_trace.out
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
open("/etc/ati/atiapfxx.blb", O_RDONLY) = 33
I have sorted out the IOMMU thing.
#if defined(CONFIG_AMD_IOMMU) || defined(CONFIG_DMAR)
#define FIREGL_DMA_REMAPPING
#endif
I have noticed CONFIG_DMAR does not exist anymore. It has been replaced by CONFIG_INTEL_IOMMU.
For people having a Intel chipset with VT-d. It might be worthwile to try. To enable FIREGL_DMA_REMAPPING,
1. replace CONFIG_DMAR by CONFIG_INTEL_IOMMU in firegl_public.c and rebuild the driver
2. Enable VT-d in BIOS
3. Edit kernel command line param in grub.cfg by adding 'intel_iommu=on'
The whole purpose FIREGL_DMA_REMAPPING is to avoid compatibility issues between different kernel version in regards to DMA API. On most distribution CONFIG_AMD_IOMMU and/or CONFIG_INTEL_IOMMU will be defined and when fglrx will be compiled, FIREGL_DMA_REMAPPING will be defined even if at runtime no HW IOMMU will be detected and it works fine anyway.
DMA API is fine. Interface is the same for HW and SW IOMMU.
DMA api ends up using functions in arch/x86/kernel/pci-nommu.c if no hardware IOMMU is on the system. So, unless proven wrong, I would say force FIREGL_DMA_REMAPPING for all setup.
That being said and after having done a fantastic journey in IOMMU and DMA kernel code, I have found out that my problem was much simpler than that:
In /etc/X11/xorg.conf, section "ServerLayout"
2 screens, each one associated to one of my 2 cards, were enumerated. I have removed the second screen of the section and by doing so, it has allowed me to activate Crossfire.
Now, X server starts but hard freeze as soon as I try to access opengl functionality. It can be something as simple as 'glxinfo'.
lano1106@whippet2 ~ $ aticonfig --adapter=0 --crossfire=on
Crossfire chain(s) enabled
CrossFire does not support on this platform
Warning: X needs to be restarted before CrossFire changes take effect.
When you get the message "CrossFire does not support on this platform", CrossFire is not enabled. I wish the error would be more verbose than that but I think that I have good lead to investigate:
lano1106@whippet2 ~ $ lsmod | grep fglrx
Module Size Used by
fglrx 5176689 203
amd_iommu_v2 7252 1 fglrx
button 4669 1 fglrx
The interesting info here is that amd_iommu_v2 is used by fglrx and
lano1106@whippet2 ~ $ dmesg | grep IOMMU
[ 2.614901] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[ 2.614902] AMD IOMMUv2 functionality not available on this system
I'll investigate the situation further but it makes sense that the AMD IOMMU does not work on my Intel board (ASUS P9X79 WS). The concept of IOMMU is totally new to me and still a bit fuzzy but apparently is provided by Intel through VT-d that you normally enable by:
and I am glad to say that I modestly contributed to its enhancement.
BFS author, Con Kolivas, has been kind enough mention my small contribution in the release entry on his blog:
http://ck-hack.blogspot.ca/2013/05/bfs-0430-ck1-for-linux-39x.html
I have been offered by the publisher to read and review this book. I was enthousiast as network programming is really something I like and I have heard about Boost.Asio usage in some projects without really having taken the opportunity to check out myself what Boost.Asio really was.
The outcome of having read it is that I was left with a lot more unanswered questions than I have received answers. It is a very short book probably of the style 'hands on' 'direct to the point' type of book which I guess has merits so someone can start reading it and hack something very fast. However, in my opinion, this is overdone making the book a bad choice for learning network programming.
My first unanswered question I had when I picked up the book is Why should I use Boost.Asio and what are its benefits over other existing frameworks? Of course, I know the answer that you can find also in the excellent book Network programming in C++ plus the fact that Boost.Asio is fully integrated with STL but the explanation is totally missing from the book.
Explanation between synchronous and asynchronous is very simplistic and can be resume as 'async is more complex than sync but eventually you will prefer async for performance reasons'. It got me a little confused for knowing that the *nix socket API and the Winsock API you can do:
blocking IO
Nonblocking IO or
async IO
which are three different ways of performing IO. I am guessing that what is really done with Boost.Asio is non blocking IO which is close to real async IO and much more common place but nowhere in the book we take the time to really explain what Boost.Asio is really doing.
Code examples are ok I guess so the author can make his point but I spotted a couple less than perfect code which I think is hurting the book credibility. I know this is a harsh judgement but for a book that aims to teach people how to program you have to be examplar as what you teach will be replicated by your readers. I am expecting perfection from a book examples. The type of code that you stare at for some time without being able to figure out how you could improve it. That is the type of quality that you'll get from a book written by Stroustrup or Stevens.
Here are two examples of what I mean:
if ( std::find(c.buff, c.buff + c.already_read, '\n') < c.buff + c.already_read) {
int pos = std::find(c.buff, c.buff + c.already_read, '\n') - c.buff;
1. It is inefficient to call twice std::find(). Imagine that '\n' is the last character in a 2GB array!
2. comparing iterators with the operator '<' works because in this case the container happens to be a POD array but IMO this is bad style to use STL algorithm in a way that will not works on all containers (ie std::list) when using the more common operator != would achieve the same result.
synchronous server code:
void handle_clients() {
while(true)
for (int i = 0; i < clients.size(); ++i)
if ( clients[i].sock.available() ) on_read(clients[i]);
}
This works but polling sockets like crazy and sucking 100% of the processor is not the way to write a server that use synchronous mode. I understand that the book did not aim to be a network programming bible like a Stevens but I would have liked some network theory background. Things such as the different possible server topologies and their benefits drawbacks of each. A very small TCP primer. How TCP connections are established from a client? from a server? I know all that but I tried to have some empathy for the young reader who haven't been exposed to this basic knowledge and since these basics are totally absent from the book, it will be even harder for him to make sense out of the book. This is the first edition so hopefully it will eventually improved.
To conclude, the book has some problems but if you can get a cheap copy it might serve as a very fast introduction to Boost.Asio and get you exposed to the API.
I want you to find in this blog informations about C++ programming that I had a hard time to find in the first place on the web.
<< | Current | >> | |
Jan | Feb | Mar | Apr |
May | Jun | Jul | Aug |
Sep | Oct | Nov | Dec |