The share of the reproduction paper is based on analyses of as well as in insights into current methods and problems—plus the certainty that is added is sold with validating past outcomes.

1. Relevance: Is this paper highly relevant to COLING?
2. Readability/clarity: could be the paper well-written and well-structured?
3. Data/code supply: may be the data/code (as appropriate) open to the investigation community or perhaps is here a compelling reason provided why this isn’t feasible?
4. Analysis: In the event that paper managed to reproduce the outcomes of the previous work, does it clearly set down exactly what would have to be filled in to carry out therefore? If it wasn’t in a position to reproduce the outcomes of earlier in the day work, does it obviously determine exactly what information had been missing/the most likely causes?
5. Generalizability: Does the paper rise above replicating the total outcomes in the initial to explore whether or not they are reproduced an additional environment? Instead, in situations of non-replicability, does the paper talk about the wider implications of the result?
6. Informativeness: To exactly what level does the analysis reported into the paper deepen our knowledge of the methodology utilized or perhaps the nagging issue approached? Will the information within the paper assistance professionals along with their range of technique/resource?
7. Significant comparison: along with pinpointing the experimental outcomes being replicated, does the paper motivate why these specific answers are a crucial target for reproduction and exactly just what the near future implications are of these having been reproduced or been discovered to be non-reproducible?
8. General recommendation: there are numerous good submissions competing for slots at COLING 2018; essential can it be to feature this 1? Will people discover a complete great deal by scanning this paper or seeing it presented? Please be decisive—it is much better to vary from other reviewers rather than grade every thing in the centre.

Site paper

Documents in this track provide a brand new language resource. This might be a corpus, but in addition could possibly be an annotation standard, device, an such like.

1. Relevance: Is this paper highly relevant to COLING? Will the resource presented likely be of good use to the community?
2. Readability/clarity: Through the means the paper is written, are you able to inform how a resource ended up being produced, the way the quality of annotations (if any) had been assessed, and exactly why the resource must be of great interest?
3. Originality: Does the resource fill a need into the current assortment of available resources? Observe that originality could possibly be into the range of language/language variety or genre, within the design associated with the annotation scheme, into the scale for the resource, or nevertheless other parameters.
4. Site quality: what type of quality control had been performed? If appropriate, ended up being inter-annotator contract calculated, if therefore, with appropriate metrics? Otherwise, the other assessment had been carried out, and exactly how acceptable were the outcome?
5. Site accessibility: could it be straightforward for researchers to down load or access that is otherwise resource to be able to make use of it in their own personal research topics for psychology work? As to what degree can work predicated on this resource be provided? answers to incorporate: Yes, i’ve confirmed
6. Metadata: perform some writers make clear whoever language use is captured when you look at the resource and also to just exactly what populations results that are experimental regarding the resource could possibly be generalized to? In the event of annotated resources, would be the demographics associated with annotators also characterized?
7. Significant contrast: may be the resource that is new with regards to current operate in the industry, including comparable resources it took motivation from or improves on? Will it be clear what is unique about the resource?
8. General recommendation: there are numerous good submissions contending for slots at COLING 2018; essential can it be to feature that one? Will people learn a complete great deal by looking over this paper or seeing it presented? Please be decisive—it is way better to change from other reviewers rather than grade every thing in the centre.

Position paper

A situation paper presents a challenge to old-fashioned thinking or even a futuristic vision that is new. It might start an area that is new unique technology, propose changes in existing research, or offer a unique group of ground guidelines.

1. Relevance: Is this paper highly relevant to COLING?
2. Readability/clarity: Is it clear just exactly what the positioning is the fact that paper is arguing for? Are the arguments because of it laid call at an understandable method?
3. Soundness: Are the arguments presented within the paper appropriate and coherent? May be the eyesight well-defined, with success requirements? (Note: it must be feasible to provide a score that is high even although you don’t concur with the place taken by the writers)
4. Imagination: How unique or bold may be the place consumed the paper? Does it express well-thought through and imaginative ground that is new?
5. Range: How scope that is much brand new scientific studies are exposed by this paper? Exactly exactly just What effect could it have on current areas and concerns?
6. Significant contrast: may be the paper well-situated with regards to past work, both place documents (taking exactly the same or opposing side for a passing fancy or similar dilemmas) and appropriate theoretical or experimental work?
7. Substance: Does the paper have sufficient substance for a paper that is full-length? Could be the presssing problem sufficiently essential? Would be the arguments adequately varied and thoughtful?
8. General suggestion: there are lots of good submissions contending for slots at COLING 2018; essential could it be to feature this 1? Please be decisive—it is way better to vary from other reviewers rather than grade every thing in the centre.

A study paper provides a structured summary of the literary works up to now on a topic that is specific assists the reader comprehend the kinds of concerns being expected about this issue, the many approaches which have been used, the way they connect with one another, and exactly just what further research areas they start. A conference-length study paper should sufficiently be about a concentrated subject so it can repeat this effectively with into the web page limits.

1. Relevance: Is the paper highly relevant to COLING?
2. Readability/clarity: could be the paper generally speaking simple to follow and well organized?
3. Organization: Does the paper arrange the appropriate literary works in a narrative and recognize typical strands of inquiry?
4. Scope: Does the paper determine a fairly concentrated area to review?
5. Thoroughness: because of the region identified to survey, does the paper address most of the appropriate literary works? May be the literature evaluated represented accurately?
6. Outlook: Does the paper determine areas for future work and/or plainly mention exactly what just isn’t yet managed in the literary works surveyed?
7. Context: Does the paper situate current research appropriately within its historic context? (We don’t expect papers to begin with Pa?ini, yet in the exact same time one thing that just cites work from 2017 most likely does not capture exactly just just how current work relates to the larger image.)
The share of the reproduction paper is based on analyses of as well as in insights into current methods and problems—plus the certainty that is added is sold with validating past outcomes.