
There is a good article Linux Kernel panic issue: How to fix hung_task_timeout_secs and blocked for more than 120 seconds problem which provides an explanation and solution to kernel hung_task_timeout_secs Issues.
By default Linux uses up to 40% of the available memory for file system caching. After this mark has been reached the file system flushes all outstanding data to disk causing all following IOs going synchronous. For flushing out this data to disk this there is a time limit of 120 seconds by default. In the case here the IO subsystem is not fast enough to flush the data withing 120 seconds. As IO subsystem responds slowly and more requests are served, System Memory gets filled up resulting in the above error, thus serving HTTP requests.
Linux Kernel panic issue: How to fix hung_task_timeout_secs and blocked for more than 120 seconds problem
Resolution
Change vm.dirty_ratio and vm.dirty_backgroud_ratio
# sysctl -w vm.dirty_ratio=10
# sysctl -w vm.dirty_background_ratio=5
# sysctl -p
If you wish to make it permanent, add the 2 lines to /etc/sysctl.conf
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10