In this case, the learning phase can make use of up to k reducers (one per fold). one aggregated classifier per fold) and then evaluated. In the case of cross-validation, the classifiers for all folds are learned in one go (i.e. The process makes use of the classifier training task to learn an aggregated classifier in one pass over the data and then evaluation proceeds in a second pass. Because Weka's Evaluation module is Aggregateable, and computes statistics incrementally, this is fairly straightforward. This task handles evaluating a classifier using either the training data, a separate test set or cross-validation. The classifier task also has various handy options such as allowing reservoir sampling to be used with batch learners (so that a maximum number of instances processed by the learning algorithm in a given map can be enforced), normal Weka filters to be used for pre-processing in each map (the task takes care of using various special subclasses of FilteredClassifier for wrapping the base classifier and filters depending on whether the base learner is Aggregateable and/or incremental), forcing batch learning for incremental learners (if desired), and for using a special "pre-constructed" filter (see below). Other, non-Aggregateable, classifiers can be combined by forming a voted ensemble using Weka's Vote meta classifier. Examples include: naive Bayes, naive Bayes multinomial, various linear regression models (learned by SGD) and Bagging. Such classifiers allow one final model, of the same type, to be produced from several separate models. Recently, a number of classifiers in Weka 3.7 have become Aggregateable. The map portion of this task can train any Weka classifier (batch or incremental) on a given data chunk and then the reduce portion will aggregate the individual models in various ways, depending on the type of classifier.
These summary statistics come in useful for some of the other tasks listed below. At the same time this task computes some handy summary statistics (that are stored as additional "meta attributes" in the header), such as count, sum, sum squared, min, max, num missing, mean, standard deviation and frequency counts for nominal values. This is particularly important because, as Weka users know, Weka is quite particular about metadata - especially when it comes to nominal attributes. Determining a unified ARFF header from separate data chunks in CSV format.In the future there could be other wrappers - one based on the Spark platform would be cool.īase map and reduce tasks distributedWekaBase version 1.0 provides tasks for: The second, called distributedWekaHadoop, provides Hadoop-specific wrappers and jobs for these base tasks. It provides base "map" and "reduce" tasks that are not tied to any specific distributed platform.
The first new package is called distributedWekaBase.
#How to give weka jar correct path series
This series of posts is continued in part 2 and part 3.
This post is the first of three that outlines what's available, in terms of distributed processing functionality, in several new packages for Weka 3.7.
#How to give weka jar correct path how to
How to handle large datasets with Weka is a question that crops up frequently on the Weka mailing list and forums.
0 kommentar(er)
