shell - How to get around the Linux "Too Many Arguments" limit -

i have pass 256kb of text argument "aws sqs" command running limit in command-line @ around 140kb. has been discussed in many places it been solved in linux kernel of 2.6.23 kernel.

but cannot work. using 3.14.48-33.39.amzn1.x86_64

here's simple example test:

#!/bin/bash  size=1000 while [ $size -lt 300000 ]    echo "$size"    var="`head -c $size < /dev/zero | tr '\0' 'a'`"    ./foo "$var"    let size="( $size * 20 ) / 19" done 

and foo script just:

#!/bin/bash echo -n "$1" | wc -c 

and output me is:

117037 123196 123196 129680 129680 136505 ./testcl: line 11: ./foo: argument list long 143689 ./testcl: line 11: ./foo: argument list long 151251 ./testcl: line 11: ./foo: argument list long 159211 

so, question how modify testcl script can pass 256kb of data? btw, have tried adding ulimit -s 65536 script , didn't help.

and if plain impossible can deal can shed light on quote link above

"while linux not plan 9, in 2.6.23 linux adding variable argument length. theoretically shouldn't hit "argument list long" errors again, patch limits maximum argument length 25% of maximum stack limit (ulimit -s)."

---

edit:

i able pass <=256kb single command line argument (see edit (4) in bottom). however, please read how did , decide if way want go. @ least should able understand why 'stuck' otherwise found out.

---

with coupling of arg_max ulim -s/4 came introduction of max_arg_strlen max. length of argument:

/*  *  linux/fs/exec.c  *  *  copyright (c) 1991, 1992  linus torvalds  */ 

...

#ifdef config_mmu /*  * nascent bprm->mm not visible until exec_mmap() can  * use lot of memory, account these pages in current->mm temporary  * oom_badness()->get_mm_rss(). once exec succeeds or fails,  * change counter via acct_arg_size(0).  */ 

...

static bool valid_arg_len(struct linux_binprm *bprm, long len) {  return len <= max_arg_strlen; } 

...

#else 

...

static bool valid_arg_len(struct linux_binprm *bprm, long len) {   return len <= bprm->p; }  #endif /* config_mmu */ 

...

static int copy_strings(int argc, struct user_arg_ptr argv,       struct linux_binprm *bprm) { 

...

    str = get_user_arg_ptr(argv, argc); 

...

    len = strnlen_user(str, max_arg_strlen);     if (!len)       goto out;      ret = -e2big;     if (!valid_arg_len(bprm, len))       goto out; 

...

} 

...

max_arg_strlen defined 32 times page size in linux/include/uapi/linux/binfmts.h:

...

/*  * these maximum length , maximum number of strings passed  * execve() system call.  max_arg_strlen random serves  * prevent kernel being unduly impacted misaddressed pointers.  * max_arg_strings chosen fit in signed 32-bit integer.  */ #define max_arg_strlen (page_size * 32) #define max_arg_strings 0x7fffffff 

...

the default page size 4kb cannot pass arguments longer 128kb.

i can't try maybe switching huge page mode (page size 4mb) if possible on system solves problem.

for more detailed information , references see this answer a similar question on unix & linux se.

---

edits:

(1) according this answer 1 can change page size of x86_64 linux 1mb enabling config_transparent_hugepage , setting config_transparent_hugepage_madvise n in kernel config.

(2) after recompiling kernel above configuration changes getconf pagesize still returns 4096. according this answer config_hugetlb_page needed pull in via config_hugetlbfs. recompiling , test again.

(3) recompiled kernel config_hugetlbfs enabled , /proc/meminfo contains corresponding hugepages_* entries mentioned in the corresponding section of kernel documentation. however, page size according getconf pagesize still unchanged. while should able request huge pages via mmap calls, kernel's default page size determining max_arg_strlen still fixed @ 4kb.

(4) modified linux/include/uapi/linux/binfmts.h #define max_arg_strlen (page_size * 64), recompiled kernel , code produces:

...

117037 123196 123196 129680 129680 136505 143689 151251 159211 

...

227982 227982 239981 239981 252611 252611 265906 ./testcl: line 11: ./foo: argument list long 279901 ./testcl: line 11: ./foo: argument list long 294632 ./testcl: line 11: ./foo: argument list long 

so limit moved 128kb 256kb expected. don't know potential side effects though. far can tell, system seems run fine.