Can't reproduce performance

[danielpol]

I'm trying to reproduce the gensort performance you mention in the Readme. By they way, it would be great to have this running multi-threaded to take advantage of multiple cores.
If I use j1e8.c I get a "Bus error" when trying to sort.
If I use bsort.c or qsort.c I run for >24 hours without finishing.
Worth to mention that when compiling bsort.c I get these warnings:
bsort.c: In function ‘shellsort’:
bsort.c:36:7: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default]
memcpy(a+j_record_size, a+(j-3)_record_size, record_size);
^
In file included from bsort.c:4:0:
/usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char _’
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,
^
bsort.c:38:5: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default]
memcpy(a+j_record_size, &temp, record_size);
^
In file included from bsort.c:4:0:
/usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char _’
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,
^
bsort.c:44:7: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default]
memcpy(a+j_record_size, a+(j-1)_record_size, record_size);
^
In file included from bsort.c:4:0:
/usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char *’
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,
^
bsort.c:46:5: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default]
memcpy(a+j_record_size, &temp, record_size);
^
In file included from bsort.c:4:0:
/usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char *’
extern void *memcpy (void **restrict __dest, const void __restrict __src,
^
bsort.c: In function ‘radixify’:
bsort.c:128:11: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default]
memcpy(&buffer[stack[stack_pointer] * record_size], &buffer[stack[stack_pointer-1] * record_size], record_size);
^
In file included from bsort.c:4:0:
/usr/include/string.h:42:14: note: expected ‘void * __restrict’ but argument is of type ‘const unsigned char *’
extern void *memcpy (void **restrict __dest, const void __restrict __src,
^
bsort.c:131:9: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default]
memcpy(&buffer[stack[0] * record_size], &temp, record_size);
^
In file included from bsort.c:4:0:
/usr/include/string.h:42:14: note: expected ‘void * __restrict’ but argument is of type ‘const unsigned char *’
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,

[adamdeprince]

Yeah, I have to clean up the warnings. What are your source files? How large are they? How much ram do you have? What is your underlying file system and I/O device? bsort is designed to perform very very well when swapping heavily, but qsort, a direct application of linux's underlying qsort function, performs very poorly when paging. j11e8 is designed for a very specific input, and deviation from that is "undefined" :-)

I may at some point make this code multithreaded - bsort is really I/O, not CPU, bound, and would benefit tremendously from having as many inflight threads as possible.

[danielpol]

I run this on output generated by gensort (100 bytes fixed binary records with 10 fixed binary key). ~10GB source files. I ended up using nsort trial version to do my testing.

[harveywi]

@danielpol, try turning on the "-v" verbose flag and seeing what happens with different values for the "-c" parameter. In my case, sorting a very small 10MB file of 128-bit numbers took over a minute, but I increased "-c" and achieved much better performance.

Some discussion about the effect of the various command line parameters would be a nice addition to the readme.

[adamdeprince]

Hi, Improvements and bug fixes for this are long overdue. Gimmie a few days and I'll get this together. - Adam

…

On Jan 22, 2019, at 3:39 PM, William Harvey ***@***.***> wrote: @danielpol <https://github.com/danielpol>, try turning on the "-v" verbose flag and seeing what happens with different values for the "-c" parameter. In my case, sorting a very small 10MB file of 128-bit numbers took over a minute, but I increased "-c" and achieved much better performance. Some discussion about the effect of the various command line parameters would be a nice addition to the readme. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC0ONfL_iLe41YY_4dNLEem_hGeGPv2ks5vF5MOgaJpZM4JmoKM>.