Server get crashed while submitting many resources.

[zyxbest]

description

These days I met the crashed again and again while uploading resource，It seems like there is something wrong with datapusher.
For test, I ran this command below:
paster --plugin=ckan datapusher submit_all -c /etc/ckan/default/development.ini
There are about 3,000 datasets in my ckan.
I also put some codes in datapusher/jobs.py to disable the datastore action:

    return  # just return !
    for i, records in enumerate(chunky(result, 250)):
        count += len(records)
        logger.info('Saving chunk {number}'.format(number=i))
        send_resource_to_datastore(resource, headers_dicts,
                                   records, api_key, ckan_url)

In the begining, all things worked right, the convert and calculation is kinda heavy but OK for my server:

During the submitting I can access the ckan website.
After about 40 minutes, the server get crashed , I cannot type any commands, the website returns timeout:

question

Why there is so high I/O waiting？
I met this crash everytime while uploading thousands of resources, even if I disable thedatastore_create (so there is nothing to do with the datastore and postgresql).

[metaodi]

The server actually downloads all the ressources, that's why you have a lot of I/O. We have similar problems and solved it by a combination of using MAX_CONTENT_LENGTH to limit DataPusher to 10MB (and #144 makes sure that the limit is actually applied to the file). So maybe step 1 would be to use the latest master, so that these limits are enforced.

The other setting that we tweaked is the datapusher apache config (https://github.com/ckan/datapusher/blob/e0319c16dcb25261359d0e9030af4dd53f94fc49/deployment/datapusher.apache2-4.conf), which uses processes=1 threads=15 in the default config. We now use processes=6 threads=15. I'm not really familiar with the details of this, but having more processes could lead to better performance especially when dealing with lots of I/O.

[zyxbest]

Thanks very much !I'm considering updating datapusher to latest!

Segmentation fault error

These days I migrate my filestore to Cloud (like AmazonS3),the I/O problem seems get improved(not very sure).
But now I face new error as 'Segmentation fault' while pushing some large files:

And then datapusher just get killed.
So is this the same problem？It performance like memory problem :

Try pushing file slowly

• When I push files one by one, (Set the datapusher job.py synchronously work , the I/O maybe low in this case), Segmentation fault still appears for large files(30M~50M)
• I've also tried to push files by 60 seconds interval, get the same result.

My server had 4G memory and 2 core CPU.Maybe I should limit the file size :(

[zyxbest]

Well, after dockerizing datapusher,and ignore some big files with the latest code, my datapusher just works fine.
Big files aslo lead to high memory use besides high I/O, then I limit the memory of datapusher container to avoid server crash.
This issue successfully helped me to solve problem!（Although the job error is still hard to catch in paster command. ）