Bug 239

Summary: Problem with multiple host threads with linux 64b driver
Product: AMD Catalystâ„¢Proprietary Display Driver Reporter: tomogenor <forum-accounts>
Component: Kernel ModuleAssignee: nobody
Severity: major    
Priority: low    
Version: .archived   
Hardware: Radeon HD 6000 Series   
OS: Linux   
Attachments: Source file for test program that produces the crash
/usr/share/fglrx/atigetsysteminfo.sh output

Description tomogenor 2011-08-25 00:42:09 CDT
Created attachment 203 [details]
Source file for test program that produces the crash

Description of problem: 
I have been developing an OpenCL program that uses multiple host threads for processing data. The multiple host threads are needed, since processing includes quite much of CPU calculation and speed is essential here.

When testing with larger number of threads the program becomes unstable -- it crashes or whole machine hangs.

So i write small test program to see if the multiple host threads are the source of the problems. The test program crashes also. On the other hand, i have windows 7 installed on this same machine. The test program works fine on windows 7 (no crash).

The test program is not suppose to do anything meaningful and some parts of the source code is copy-pasted from real library, that i cannot publish sources for (if you wonder why thing X is done in way Y).

The kernel source code and copyright notice are from matrix multiplication example provided in SDK 2.5.

What happens with linux is that program runs fine with some amount of threads and then crashes. The core dump looks like this:

#0  0x00007f5f9c095917 in ?? () from /usr/local/lib/libamdocl64.so
#1  0x00007f5f9c05aa0c in clEnqueueNDRangeKernel () from /usr/local/lib/libamdocl64.so
#2  0x0000000000401f10 in benchmark_run_single_test (status=0x7fffea3c3f90, queue_id=0,
    thread_id=11, print_enable=true) at main.c:537
#3  0x00000000004021e2 in benchmark_run_single_test_threaded (void_param=0x6e60538)
    at main.c:595
#4  0x00007f5fa0880d8c in start_thread (arg=0x7f5f79ffb700) at pthread_create.c:304
#5  0x00007f5fa0b7e04d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Steps to reproduce:
1. Compile program (main.c is only source, requires libOpenCL and libpthread)
2. Run the program ./opencl_test
3. Observer crash, usually after some amounts of threads running

I am running "Linux lant 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux"

Here is clinfo outputs:

Number of platforms:                             1
  Platform Profile:                              FULL_PROFILE
  Platform Version:                              OpenCL 1.1 AMD-APP-SDK-v2.5 (684.213)
  Platform Name:                                 AMD Accelerated Parallel Processing
  Platform Vendor:                               Advanced Micro Devices, Inc.
  Platform Extensions:                           cl_khr_icd cl_amd_event_callback cl_amd_offline_devices

  Platform Name:                                 AMD Accelerated Parallel Processing
Number of devices:                               2
  Device Type:                                   CL_DEVICE_TYPE_GPU
  Device ID:                                     4098
  Device Topology:                               PCI[ B#6, D#0, F#0 ]
  Max compute units:                             12
  Max work items dimensions:                     3
    Max work items[0]:                           256
    Max work items[1]:                           256
    Max work items[2]:                           256
  Max work group size:                           256
  Preferred vector width char:                   16
  Preferred vector width short:                  8
  Preferred vector width int:                    4
  Preferred vector width long:                   2
  Preferred vector width float:                  4
  Preferred vector width double:                 0
  Native vector width char:                      16
  Native vector width short:                     8
  Native vector width int:                       4
  Native vector width long:                      2
  Native vector width float:                     4
  Native vector width double:                    0
  Max clock frequency:                           790Mhz
  Address bits:                                  32
  Max memory allocation:                         134217728
  Image support:                                 Yes
  Max number of images read arguments:           128
  Max number of images write arguments:          8
  Max image 2D width:                            8192
  Max image 2D height:                           8192
  Max image 3D width:                            2048
  Max image 3D height:                           2048
  Max image 3D depth:                            2048
  Max samplers within kernel:                    16
  Max size of kernel argument:                   1024
  Alignment (bits) of base address:              32768
  Minimum alignment (bytes) for any datatype:    128
  Single precision floating point capability
    Denorms:                                     No
    Quiet NaNs:                                  Yes
    Round to nearest even:                       Yes
    Round to zero:                               Yes
    Round to +ve and infinity:                   Yes
    IEEE754-2008 fused multiply-add:             Yes
  Cache type:                                    None
  Cache line size:                               0
  Cache size:                                    0
  Global memory size:                            536870912
  Constant buffer size:                          65536
  Max number of constant args:                   8
  Local memory type:                             Scratchpad
  Local memory size:                             32768
  Kernel Preferred work group size multiple:     64
  Error correction support:                      0
  Unified memory for Host and Device:            0
  Profiling timer resolution:                    1
  Device endianess:                              Little
  Available:                                     Yes
  Compiler available:                            Yes
  Execution capabilities:                               
    Execute OpenCL kernels:                      Yes
    Execute native function:                     No
  Queue properties:                             
    Out-of-Order:                                No
    Profiling :                                  Yes
  Platform ID:                                   0x7f8fb8b82060
  Name:                                          Barts
  Vendor:                                        Advanced Micro Devices, Inc.
  Device OpenCL C version:                       OpenCL C 1.1
  Driver version:                                CAL 1.4.1457
  Profile:                                       FULL_PROFILE
  Version:                                       OpenCL 1.1 AMD-APP-SDK-v2.5 (684.213)
  Extensions:                                    cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_ext_atomic_counters_32 cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_popcnt

I tried running with CPU device. The CPU code works fine (no crash). After changing the parameters (matrix size, memory size, image size) the same code that runned fine with CPU -device execution crashes still with GPU execution, though with different thread amount than earlier (now with 28 host threads).

By commenting out the kernel code (clEnqueueNDRangeKernel) i get 'CL_OUT_OF_RESOURCES' (on clEnqueueReadBuffer) but no crash. After that there is only calls: clEnqueueWriteImage -> clWaitForEvents -> clEnqueueReadBuffer -> clWaitForEvents.

Seems like matrix size has little effect (memory size 5*(1<<20))
With matrix size 64 -- crash at 28 threads (CPU runs fine)
With matrix size 128 -- crash at 28 threads (Not tested with Cpu)
With matrix size 256 -- crash at 28 threads (Not tested with Cpu)


On the other hand, seems like memory size does affect (matrix size 64):
Memory size (1<<20) -> no crash
Memory size 10*(1<<20) -> crash at 14 threads (Not tested with Cpu)
Memory size 20*(1<<20) -> crash at 8 threads (CPU runs fine)
Comment 1 tomogenor 2011-08-25 00:44:20 CDT
Created attachment 204 [details]
/usr/share/fglrx/atigetsysteminfo.sh output
Comment 2 Michael Cronenworth 2012-03-08 21:18:50 CST
This message is a reminder that your bug is marked as Catalyst 11.8.

The current Catalyst version is 12.2.

Approximately 7 days from now the Bugzilla administrator will be removing the
11.8 version. At that time your bug will be CLOSED as WONTFIX.

Bug Reporter: Thank you for reporting this issue. However, the Bugzilla
administrator provides this as a unofficial, free service to AMD customers, and
I like to keep my systems neat and tidy. If you would like to keep your bug
from being closed, please try a new Catalyst version and update the 'version'
field if the issue still occurs.

If you are unable to update the version, please make a comment and someone will
change it for you.
Comment 3 Michael Cronenworth 2012-03-15 23:30:41 CDT
This bug is being closed due to the 'version' being 11.8 after 7 days of the
previous closure notice.

Thank you for your bug report.