machine learning - Why does RSquared increase with # of folds in k-fold cross validation? -

-

i'm tuning model using k-fold cross-validation , noticed rsquared accuracy appears improve number of folds -- e.g. higher rsquared value when using 30 folds compared using 10 folds.

two questions hoping insight on:

  1. why occur?
  2. is there reason believe rsquared k=10 better estimate of model accuracy using k=30? or both unrelated future error rate can expect on unseen test set?

here's simple example of effect i'm referring to:

############### k = 10 ##################### > data(iris) > train_control <- traincontrol(method="repeatedcv", number=10, repeats=3) > train(sepal.length~.,data=iris,trcontrol=train_control,method="rf",metric="rsquared")  random forest   150 samples 4 predictor  no pre-processing resampling: cross-validated (10 fold, repeated 3 times)  summary of sample sizes: 137, 135, 134, 134, 135, 136, ...  resampling results across tuning parameters:  mtry  rmse       rsquared   rmse sd     rsquared sd 2     0.3381065  0.8404534  0.07692415  0.07583768  3     0.3247406  0.8502577  0.07311807  0.07326181  5     0.3228651  0.8517740  0.07213958  0.07315720   ############### k = 30 ##################### > data(iris) > train_control <- traincontrol(method="repeatedcv", number=30, repeats=3) > train(sepal.length~.,data=iris,trcontrol=train_control,method="rf",metric="rsquared")  random forest   150 samples 4 predictor  no pre-processing resampling: cross-validated (30 fold, repeated 3 times)  summary of sample sizes: 143, 145, 146, 144, 145, 144, ...  resampling results across tuning parameters:  mtry  rmse       rsquared   rmse sd     rsquared sd 2     0.3238545  0.8580474  0.10327919  0.1352787   3     0.3119541  0.8679321  0.09734168  0.1236307   5     0.3109572  0.8717550  0.09727307  0.1123173

bigger number of folds - bigger training set , smaller testing one. observe best result fo loo (n-fold n training samples) , worst k=2. there no 1 answer generic question how many folds use, solely depends on dataset. furthermore if there underlying relation between datapoints (for example come time series) important how set divided.


Comments

Post a Comment

Popular posts from this blog

How to show in django cms breadcrumbs full path? -

-
need advice - how show in breadcrumbs whole path article or page? i've got code: {% ance in ancestors %} <li> {% if not forloop.last %} <a href="{{ ance.get_absolute_url }}">{{ ance.get_menu_title }}</a> {% else %} <span class="active">{{ ance.get_menu_title }}</span> {% endif %} </li> {% endfor %} but shows "home" , current page. how modify show whole path in breadcrumbs? i've found in file menu_tags.py code calls ancestors. @ least there class showbreadcrumbs: class showbreadcrumb(inclusiontag): """ shows breadcrumb node has same url current request - start level: after level should breadcrumb start? 0=home - template: template used render breadcrumb """ name = 'show_breadcrumb' template = 'menu/dummy.html' options = options( argument('start_level', default=0, require...
Read more

php - Invalid Cofiguration - yii\base\InvalidConfigException - Yii2 -

-
Image
i've installed yii2-user module/dektrium yii2-app-basic application command composer require "dektrium/yii2-user:0.9.*@dev" config/console.php return [ . . 'modules' => [ 'gii' => 'yii\gii\module', 'user' => [ 'class' => 'dektrium\user\module', ], ], . . ] config/web.php 'components' => [ . . /* 'user' => [ 'identityclass' => 'app\models\user', 'enableautologin' => true, ], */ 'modules' => [ 'user' => [ 'class' => 'dektrium\user\module', ], ], . . ] after that, run command $ php yii migrate/up --migrationpath=@vendor/dektrium/yii2-user/migrations updating database schema . but, when run http://localhost/mylawsuit/yii/web/index.php?r=u...
Read more

ruby on rails - npm error: tunneling socket could not be established, cause=connect ETIMEDOUT -

-
i have app uses rails framework , implements angularjs part of front end. i have pushed heroku , have heroku toolbelt installed, when try migrate db using "heroku run rake db:migrate" receive following error(s): installing core plugins heroku-cli-addons, heroku-apps, heroku-fork, heroku-git, heroku-local, heroku-run, heroku-status... error installing package. try running again gode_debug=info see more output. ! `run` not heroku command. ! perhaps meant `-h`, `2fa`, `auth`, `join`, `open`, `orgs`, `pg`, `ps` or `rake`. ! see `heroku help` list of available commands. i run command "gode_debug=info heroku run rake db:migrate" , receive error: npm err! darwin 14.5.0 npm err! argv "/users/christopher_pelnar/.heroku/node-v4.2.1-darwin-x64/bin/node" "/users/christopher_pelnar/.heroku/node-v4.2.1-darwin-x64/lib/node_modules/npm/cli.js" "install" "heroku-cli-addons" "heroku-apps" "heroku-fork...
Read more